Commerce Recommendations: Technical Implementation Reference

Document type: implementation-as-built technical reference
System version: package 0.1.0 / HTTP API 1.0.0
Baseline source review: 2026-09-21; architecture, capability, and publication reconciliation 2026-09-25 Primary audience: application engineers, data engineers, operators, reviewers, and API consumers

This document explains the complete implemented service: process boundaries, domain rules, source contracts, recommendation algorithms, evaluation, persistence, concurrency, publication, serving, failure handling, configuration, operations, and verification. Diagrams use Mermaid and are stored beside the code so they evolve with it.

For the reasons behind the mechanisms, their costs, and current limitations, read engineering rationale. For contract-to-code change recipes, read implementation workflow. Historical baseline diagrams are simplified; the current module table and additive-capability sections include newer paths.

Contents: documentation and authority · boundaries · architecture · entry points · domain · configuration · HTTP API · source contract · training pipeline · algorithms · evaluation · persistence · lifecycle · failures · retention · security · operations · verification · modules · guarantees and limits · additive capabilities

1. Documentation map and authority

Document	Purpose	Authority
PRODUCT_VISION_STRATEGY.md	Product vision, strategy, evidence, and decisions	Approved upstream intent
CONTEXT.md	Canonical product language	Terminology source
specs/commerce-recommendation-service.md	Observable requirements and acceptance examples	Approved behavioral specification
design/commerce-recommendation-service.md	Pre-implementation design and test plan	Approved design baseline
README.md	Setup, invocation, and operator quick start	Operational entry point
This document	Current implementation and diagrams	Code-aligned technical reference

When documents disagree, use the approved specification for intended product behavior and the current source code plus migration for deployed behavior. This reference deliberately describes the latter and labels important qualification boundaries.

2. System purpose and boundaries

The service builds property-local, catalog-local recommendation snapshots from configured relational catalog, view, and purchase data. Training is asynchronous. Serving reads published snapshot artifacts and authorized personalization projections; it never contacts the merchant source or trains synchronously. Default-off personalization can rerank bounded candidates from that same snapshot using a service-owned Shopper Profile; it does not alter snapshots or read the source while serving. Training stages snapshot-scoped candidate category, brand, and Recommendation Eligibility from its consistent Catalog read before activating the serving head. These companion rows use bounded publication batches and are removed on publication failure, abandoned-build replacement, or snapshot retention. Older snapshots without feature rows retain the missing-metadata fallback.

The isolation key is the triple:

(data_source_id, tracking_id, catalog_id)

Every run, source read, snapshot, serving head, and API lookup is bound to this scope. Product IDs are meaningful only inside it.

Implemented recommendation strategies:

Strategy	Shape	Primary signal	Ordered fallback tiers
Frequently Bought Together	Anchored	Same-order co-purchase	Category-pair purchase, compatibility rule, category purchase popularity, property purchase popularity
Also Viewed	Anchored	Same-session co-view	Metadata similarity, category view popularity, property view popularity
Similar Items	Anchored	Metadata cosine	None
Best Sellers	Global	Time-decayed purchase quantity	None
Most Viewed	Global	Time-decayed view count	None
Trending	Global	Positive recent view-and-purchase momentum	None
Popular Viewed Same Category	Anchored	Time-decayed views in the anchor's known category	None
Popular Bought Same Category	Anchored	Time-decayed purchase quantity in the anchor's known category	None
Frequent Bought Different Category	Anchored	Supported same-order co-purchase across known categories	None
Frequent Viewed Same Category	Anchored	Supported same-session co-view within a known category	None
Frequent Viewed Different Category	Anchored	Supported same-session co-view across known categories	None

Category-constrained strategies retain the corresponding popularity or co-occurrence provenance. They exclude missing categories and do not widen to unrelated evidence when a lane is empty. For behavioral lanes, category relations are applied in the derived work store before the bounded per-anchor top-k selection. Holdout truth is restricted to the same relation for evaluation.

When compatibility_query is configured, the source adapter streams property/catalog-scoped product-to-product complement rules. The pipeline ranks those eligible rule candidates by purchase popularity, with stable Product ID ties, and drains them before category/property popularity.

Out of scope in the current implementation:

• Cross-property or cross-catalog evidence.
• Source schema management or arbitrary query submission through the API.
• Scheduling; an external scheduler submits training runs.
• General authentication and authorization for the pre-existing training and ordinary serving APIs. The additive personalization interfaces require property-bound authentication and signed authorization context as documented in shopper-personalization.md.
• Synchronous training during a serving request. Optional reranking, recent-seed fusion, and snapshot ANN retrieval do perform bounded request-time computation.
• A distributed training engine.
• Production authorization for the ordinary training and non-personalized serving surfaces; deployment/network controls still own that boundary.
• Claims that synthetic simulation establishes production scale, causal uplift, revenue, or conversion impact.

3. Architecture

3.1 System context

flowchart LR
    scheduler[External scheduler or training caller]
    consumer[Serving consumer]
    property[Authenticated Commerce Property integration]
    api[Recommendations API process]
    worker[Recommendations worker process]
    control[(Service-owned PostgreSQL)]
    source[(Configured relational source)]
    scratch[(Ephemeral DuckDB and spill directory)]
    collector[OTLP Collector]

    scheduler -->|POST or GET training runs| api
    consumer -->|GET recommendation sets| api
    property -->|Interactions and personalized reads| api
    api -->|Run coordination and snapshot reads| control
    api -->|Profiles, ledgers, suppression| control
    worker -->|Claim, lease, publish, retain| control
    worker -->|Parameterized consistent streams| source
    worker -->|Derived aggregates only| scratch
    api -. enabled telemetry .-> collector
    worker -. enabled telemetry .-> collector

3.2 Component responsibilities

flowchart TB
    subgraph API_Process[API process]
        routes[FastAPI routes]
        contracts[Pydantic contracts]
    end

    subgraph Worker_Process[Worker process]
        coordinator[TrainingWorker]
        adapter[SQLAlchemy source adapter]
        pipeline[Pipeline orchestrator]
        content[Content similarity]
        behavioral[Behavioral co-occurrence]
        popularity[Popularity and trending]
        evaluation[Temporal evaluation]
        ranker[Tiered candidate ranker]
        workstore[DerivedWorkStore]
    end

    subgraph Shared[Shared package]
        domain[Domain values and invariants]
        repositories[Training and snapshot repositories]
        config[Deployment configuration]
    end

    routes --> contracts
    routes --> repositories
    routes --> domain
    coordinator --> repositories
    coordinator --> adapter
    coordinator --> pipeline
    pipeline --> content
    pipeline --> behavioral
    pipeline --> popularity
    pipeline --> evaluation
    pipeline --> ranker
    pipeline --> workstore
    adapter --> config
    contracts --> domain
    repositories --> domain

3.3 Deployment view

flowchart LR
    subgraph Service_Runtime[Service runtime]
        api1[API replica]
        api2[API replica]
        worker1[Worker]
        worker2[Worker]
    end

    postgres[(PostgreSQL control and snapshots)]
    merchantA[(Merchant source A)]
    merchantB[(Merchant source B)]
    local1[(Worker-local derived scratch)]
    local2[(Worker-local derived scratch)]

    api1 --> postgres
    api2 --> postgres
    worker1 --> postgres
    worker2 --> postgres
    worker1 --> merchantA
    worker2 --> merchantB
    worker1 --> local1
    worker2 --> local2

API and worker entry points are separate console scripts. Multiple workers may operate in parallel on independent scopes. Database row locking and a partial unique index coordinate work; SQLite is only a local/test substitute and does not establish the production concurrency claim. The Docker Compose stack adds a separate Simulation Data Source, storefront, and one-shot visitor runner; those components exercise the public service boundaries and never write snapshots directly.

4. Runtime entry points and dependency assembly

Entry point	Function	Behavior
recommendations-api	recommendations.api:main	Loads deployment config, creates the service store and app, then starts Uvicorn on RECOMMENDATIONS_API_HOST/RECOMMENDATIONS_API_PORT (defaults 0.0.0.0:8000)
recommendations-worker	recommendations.worker:main	Loads config, creates one SQLAlchemy engine per source, creates the service store, and polls indefinitely
recommendations-generate-source	recommendations.synthetic.cli:main	Generates a deterministic SQLite or PostgreSQL relational source plus lineage, oracle, and verified materialization artifacts; PostgreSQL streams through bounded parallel COPY workers and atomically publishes verified staging tables
recommendations-simulation	recommendations.simulation.cli:main	Resolves and runs a bounded local/test Simulation Scenario through storefront HTTP
main.py	Compatibility wrapper	Calls the API main function
alembic	migrations/env.py	Applies the service-owned schema
scripts/export_openapi.py	Schema export	Writes the generated OpenAPI contract
scripts/qualify.py	Qualification client	Measures the approved capacity and API latency objectives against a running environment

Dependency assembly occurs at process entry points. create_source_adapters is the worker's composition-root factory; inner code depends on the SourceAdapter and SourceRead protocols. The API composition root optionally assembles admission control, observability, and personalization services from validated deployment configuration. Tests use create_app, injected clocks and lifecycle policies, create_sqlite_store, and explicit adapters, which keeps behavior deterministic.

Ordinary and personalized serving decisions live in RecommendationServingService. Its SnapshotReader seam exposes published snapshot reads only. SnapshotLaneLoader and ForYouCandidateLoader supply bounded candidate bundles; an assembled AuthorizedPersonalization or UnavailablePersonalization policy handles the personalization attempt. The HTTP adapter maps typed failures to existing status codes and telemetry outcomes. Application tests can exercise scope rejection, freshness, missing results, fallback, and causal failures without FastAPI or SQL. Configured Swimlane resolution separately compiles each step once into its required variations and candidate behavior, then loads those variations from one head before ordered composition.

For a PostgreSQL Generated Relational Data Source, the CLI calls provision_postgresql with a GenerationConfig. The generator defers its manifest, then streams each canonical table once in Catalog-first order. One producer computes the per-table count and SHA3-512 digest while feeding bounded queues to up to 16 psycopg COPY workers. After the workers commit, the generator completes the same content-derived manifest used by the eager SQLite path. The materializer builds canonical-order indexes, independently reads the staging tables to recompute counts and digests, checks relational constraints, and renames the verified tables in one publication transaction. The CLI writes the manifest, oracle, and receipt artifacts after publication. Failure before that transaction leaves the prior published source intact. The dedicated compose.generated-source.yaml stack keeps this provisioning database separate from the mutable Simulation Data Source.

The same provision_postgresql entry point accepts an append_to Generation Manifest. GeneratedRelationalDataSource.extend resolves a bounded interval with fresh context identities, continuing row sequences and the base profile's daily rates while preserving Catalog metadata. The PostgreSQL materializer verifies the locked parent, streams new interactions through one transactional COPY writer, and verifies the complete resulting source using the same canonical digest reader as creation/replacement. Rows and generator-owned lineage commit together. Its recommendations_generated_source_state table records the latest aggregate manifest, allowing an exact-cutoff retry to repair artifact export without duplicating activity. Replacement clears that lineage. Shared GeneratedSourceEvidence and MaterializedSource writers export manifests and the standard receipt atomically per file. See the append contract for interval limits, recovery, and the additional full-source verification cost.

Generator v3 derives group-count estimates from the actual repeating size distributions. This corrects the v2 purchase estimate that could fall back to an impossible group larger than the Catalog. Regular purchase groups stay at or below 200 distinct Items, alongside the intentional 201- and 400-Item test groups, while preserving exact row and repeat quotas. Affected generated datasets change identity; RNG seeding and the manifest schema remain unchanged. Generation is still serial, with parallelism confined to the materializer's COPY, indexing, and verification.

5. Domain model

5.1 Core class diagram

classDiagram
    class CommerceScope {
        +str data_source_id
        +str tracking_id
        +str catalog_id
    }

    class TrainingRun {
        +str run_id
        +CommerceScope scope
        +str idempotency_key
        +TrainingRunStatus status
        +datetime requested_at
        +datetime started_at
        +datetime finished_at
        +datetime data_cutoff
        +str snapshot_id
        +str failure_category
        +dict metrics
        +str metrics_schema_version
        +str config_version
        +str model_version
        +tuple state_transitions
    }

    class RecommendationSnapshot {
        +str snapshot_id
        +CommerceScope scope
        +datetime generated_at
        +datetime data_cutoff
        +int freshness_hours
    }

    class RecommendationSet {
        +RecommendationStrategy strategy
        +str anchor_product_id
        +RecommendationSetOutcome outcome
        +tuple entries
        +post_init()
    }

    class RankedRecommendation {
        +str product_id
        +int rank
        +float score
        +ConfidenceTier confidence_tier
        +RecommendationProvenance provenance
    }

    class RecommendationProvenance {
        +str source_tier
        +EvidenceType evidence_type
        +int support
        +str fallback_reason
    }

    class RecommendationCandidate {
        +str product_id
        +float score
        +EvidenceType evidence_type
        +int support
    }

    CommerceScope "1" <-- "0..*" TrainingRun : scopes
    CommerceScope "1" <-- "0..*" RecommendationSnapshot : scopes
    RecommendationSnapshot "1" o-- "0..*" RecommendationSet : contains
    RecommendationSet "1" *-- "0..100" RankedRecommendation : entries
    RankedRecommendation "1" *-- "1" RecommendationProvenance : explains
    RecommendationCandidate ..> RankedRecommendation : ranked into

5.2 Domain invariants

Identifiers are normalized to Unicode NFC and must contain 1 through 255 UTF-8 bytes. Timestamps are normalized to UTC at persistence and source boundaries.

RecommendationSet validates the stored and served ranking artifact:

• Best Sellers, Most Viewed, and Trending must not have an anchor.
• Frequently Bought Together, Also Viewed, Similar Items, and the five category-constrained strategies must have an anchor.
• A set contains at most 100 entries.
• Ranks are contiguous and one-based.
• Product IDs are unique within a set.
• An anchored set cannot recommend its anchor.
• An insufficient_evidence set must be empty.

Candidate ranking additionally removes ineligible products, self-recommendations, and duplicates. Earlier evidence tiers always win over later tiers, regardless of raw score scale. Inside one tier, candidates sort by descending score and then ascending Product ID.

Confidence assignment is explanatory, not a re-ranking input:

• high: co-purchase or co-view support is at least 10 by default;
• medium: other behavioral, metadata, category-pair, or compatibility evidence;
• low: category/property popularity and all global popularity/momentum evidence.

6. Configuration model

load_deployment_config acquires the environment-selected JSON document and delegates to DeploymentConfig.from_mapping. Configuration value types own section-level from_mapping factories for source scopes, sources, pipeline, freshness, retention, personalization principals, personalization policy, and Swimlanes. Existing strict validators and defaults are retained; deployment assembly validates references between sections. Section factories do not resolve credentials. Observability parsing continues to apply its documented environment allowlist.

6.1 Configuration class diagram

classDiagram
    class DeploymentConfig {
        +str control_database_url
        +DataSourceRegistry data_sources
        +float worker_poll_seconds
        +PipelineConfig pipeline
        +FreshnessConfig freshness
        +RetentionConfig retention
        +QualityDriftConfig quality_drift
        +ObservabilityConfig observability
        +PersonalizationConfig personalization
    }

    class PipelineConfig {
        +str memory_limit
        +str max_temp_directory_size
        +int threads
        +int batch_key_limit
        +CooccurrenceBackend cooccurrence_backend
        +int view_session_distinct_product_limit
        +int purchase_order_distinct_product_limit
    }

    class DataSourceRegistry {
        -dict configurations_by_id
        +contains(data_source_id) bool
        +get(data_source_id) DataSourceConfig
        +configurations() tuple
    }

    class DataSourceConfig {
        +str data_source_id
        +str connection_url_env
        +str catalog_query
        +str views_query
        +str purchases_query
        +str offline_purchases_query
        +str compatibility_query
        +str isolation_level
        +tuple scopes
        +canonical_queries() CanonicalQueries
        +connection_url() str
    }

    class CanonicalQueries {
        +str catalog
        +str views
        +str purchases
        +str offline_purchases
        +str compatibility
    }

    class PersonalizationConfig {
        +tuple capabilities
        +tuple principals
        +frozenset allowed_policy_versions
        +str signing_secret_env
    }

    class ObservabilityConfig {
        +bool enabled
        +str deployment_environment
        +CollectorConfig collector
        +SignalConfig signals
        +TelemetryLimits limits
    }

    DeploymentConfig *-- PipelineConfig
    DeploymentConfig *-- FreshnessConfig
    DeploymentConfig *-- RetentionConfig
    DeploymentConfig *-- QualityDriftConfig
    DeploymentConfig *-- ObservabilityConfig
    DeploymentConfig *-- PersonalizationConfig
    DeploymentConfig *-- DataSourceRegistry
    DataSourceRegistry *-- "1..*" DataSourceConfig
    DataSourceConfig ..> CanonicalQueries : creates

6.2 Environment variables

Variable	Required by	Meaning
RECOMMENDATIONS_CONTROL_DATABASE_URL	API and worker	SQLAlchemy URL for the service-owned database
RECOMMENDATIONS_DATA_SOURCES_FILE	API and worker	Path to the JSON deployment file
Value named by each connection_url_env	Worker	Merchant-source SQLAlchemy URL; the API only needs the source identity registry
Value named by personalization.signing_secret_env	API when any capability is enabled	HMAC secret for signed context and causal tokens; minimum 32 bytes
Value named by each personalization principal credential_env	API when any capability is enabled	Opaque bearer credential resolved to a property-scoped principal
RECOMMENDATIONS_OBSERVABILITY_ENABLED and RECOMMENDATIONS_DEPLOYMENT_ENVIRONMENT	API and worker	Allowlisted overrides for telemetry activation and environment safety policy
OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_EXPORTER_OTLP_CERTIFICATE	API and worker when telemetry is enabled	Allowlisted Collector transport overrides

Secrets are indirect: the JSON file contains environment-variable or file references, not a credential value. Configuration hashing and representations exclude resolved secrets.

6.3 Deployment JSON

Top-level fields:

Field	Default	Validation
worker_poll_seconds	1.0	Numeric and greater than zero
pipeline.memory_limit	4GB	Non-empty DuckDB value
pipeline.max_temp_directory_size	100GB	Non-empty DuckDB value
pipeline.threads	4	Positive native-compute budget; used by DuckDB reduction/query work and metadata sparse multiplication
pipeline.batch_key_limit	100000	Positive per-batch aggregate-key/contribution bound; see section 9.3
pipeline.cooccurrence_backend	duckdb	duckdb (native SQL), sparse (SciPy CSR), accelerate (macOS BLAS), torch (MPS GPU), mlx (Metal GPU), or ane (ANEForge Neural Engine); included in the configuration fingerprint
pipeline.view_session_distinct_product_limit	100	Positive integer
pipeline.purchase_order_distinct_product_limit	200	Positive integer
pipeline.min_support_grid	2, 3, 5	Unique positive integers
pipeline.shrinkage_grid	5, 10, 20	Unique finite positive numbers
pipeline.view_half_life_days_grid	3, 7, 14	Unique finite positive numbers
pipeline.purchase_half_life_days_grid	14, 30, 60	Unique finite positive numbers
pipeline.trending_recent_days_grid	3, 7, 14	Unique positive integers
pipeline.trending_baseline_days_grid	14, 28, 56	Unique positive integers
pipeline.trending_alpha_grid	0.01, 0.1, 1	Unique finite positive numbers
pipeline.trending_view_weight	1	Finite and non-negative; at least one trend weight is positive
pipeline.trending_purchase_weight	1	Finite and non-negative; at least one trend weight is positive
freshness.default_hours	48	Positive integer
freshness.overrides	empty	Unique property/catalog scopes with positive hours
retention.snapshots_per_scope	5	Positive integer
retention.training_run_days	90	Positive integer
retention.failed_diagnostic_days	30	Positive integer
quality_drift.enabled	false	Enables bounded same-scope, same-version diagnostic comparison; never blocks publication
observability.enabled	false	Enables the sibling telemetry runtime; production-like insecure OTLP transport is rejected
personalization.capabilities	empty	Exact scope/version allowlist; every entry is default-off unless enabled explicitly
personalization.principals	empty	Env-backed property principals with non-empty role sets
data_sources	none	Required list of objects

Every data-source object requires data_source_id, connection_url_env, a non-empty list of unique tracking/catalog scopes, catalog_query, views_query, and purchases_query. isolation_level, offline_purchases_query, and compatibility_query are optional. The online and offline purchase streams are read separately and the internal group key includes a channel prefix, so identical Order IDs in the two channels do not merge. Compatibility rows contain tracking_id, catalog_id, anchor_product_id, and product_id.

The loader rejects unknown keys at every level. Its canonical SHA3-512 config_version covers source mappings, Pipeline configuration, freshness, retention, quality drift, and secret-free personalization configuration while excluding resolved credentials and runtime-only observability values. Personalization is only assembled when at least one exact-scope capability is enabled; enabled configuration also requires principals, accepted policy versions, and a signing-secret reference.

7. HTTP API

7.1 Endpoint summary

Method and path	Purpose	Success
POST /v1/training-runs	Submit an asynchronous scoped run	202 with the pending run, or the existing idempotent run
GET /v1/training-runs/{run_id}	Inspect lifecycle, metrics, and outcome	200
POST /v1/personalization/interactions	Append one authorized, ordered, idempotent Shopper interaction	202 with committed sequence, profile version, and causal token
GET /v1/catalogs/{catalog_id}/items/{product_id}/recommendations/{strategy}	Serve an anchored strategy	200
GET /v1/catalogs/{catalog_id}/recommendations/{strategy}	Serve a global strategy, including personalized for-you	200
POST /v1/catalogs/{catalog_id}/swimlanes/{swimlane_name}:resolve	Read one configured, property-scoped Swimlane from a single published Snapshot	200

Training submission accepts exactly data_source_id, tracking_id, and catalog_id. It requires an Idempotency-Key header of 1 through 128 characters. Request bodies reject extra fields. Source SQL, credentials, tuning parameters, and scheduling instructions cannot enter through the API.

Serving requires data_source_id and tracking_id query parameters. limit defaults to 20 and must be between 1 and 100. Without personalization, the service slices the persisted list and does not rerank or recompute scores. Existing strategies opt in to bounded reranking with personalize=true; for-you implies personalization. Both use Authorization, Personalization-Context, optional Causal-Token, and optional Personalization-Version headers. An unauthorized personalized read falls back to a non-personalized order with explicit response metadata rather than disclosing why authorization failed.

The contextual Swimlane POST is read-only and requires an authenticated property bearer with the personalization:serve role before parsing its JSON body. Its Commerce Scope comes from that principal and the catalog path, not from caller-supplied data-source or tracking fields. The body accepts an optional anchor_product_id, an optional quota no greater than the configured quota, and optional userContext with a complete country/region/city hierarchy. A configured sequence tries each strategy/geographic variation in order, deduplicates Items, and stops at the quota. An optional authorized Personalization-Context enables typed recent-item strategies and the terminal eligible recent-view history filler; an unauthenticated or unavailable profile is never read. Every selected Item reports its actual step and geographic level, but no local support or score. Responses use Cache-Control: private, no-store. Existing GET routes remain global.

The derived work store expands a validated source context into city, enclosing region, and country partitions while retaining its existing global aggregate. It writes bounded pair, support, and daily-popularity deltas without storing session or order IDs. Geographic publication is sparse: at most 512 local partitions and 50,000 local sets are considered per run, and an Item must have support from at least ten distinct contexts at its level to enter a local set. Local sets contain only local behavioral evidence; the Swimlane, not training, performs cross-level fallback.

The shipped config/data_sources.example.json defines four lanes for its example Commerce Scope. They do not change the legacy GET routes:

Lane	Request context	Ordered configured steps before the terminal history filler
home-discovery	No anchor required	Global Trending → Best Sellers → Most Viewed
item-discovery	Item anchor; geography optional	Also Viewed city → region → country → global → Similar Items global → Best Sellers global
item-complements	Item anchor; geography optional	Frequently Bought Together city → region → country → global → Frequent Bought in a Different Category global → Best Sellers global
category-explore	Item anchor with a known category; geography optional	Frequent Viewed in the Same Category city → region → country → global → Popular Viewed in the Same Category global → Similar Items global → Most Viewed global

For example, item-discovery is configured as:

{
  "data_source_id": "merchant-platform",
  "tracking_id": "store-42",
  "catalog_id": "main",
  "name": "item-discovery",
  "quota": 20,
  "steps": [
    {"strategy": "also-viewed", "geography": "city"},
    {"strategy": "also-viewed", "geography": "region"},
    {"strategy": "also-viewed", "geography": "country"},
    {"strategy": "also-viewed", "geography": "global"},
    {"strategy": "similar-items", "geography": "global"},
    {"strategy": "best-sellers", "geography": "global"}
  ]
}

This object belongs in the deployment's top-level swimlanes list and needs an existing scope and property-serving principal. With the shipped example, the API also needs values for STORE_42_PERSONALIZATION_TOKEN and RECOMMENDATIONS_PERSONALIZATION_SIGNING_SECRET (at least 32 bytes), supplied through the deployment's secret system rather than committed files. The example is not loaded by the default simulation Compose stack, and the application does not invent a Swimlane when configuration omits one. Missing local geography skips that variation; the explicit global steps still work. Item-anchored steps also skip when no anchor is supplied. If configured steps underfill, eligible authorized recent browsing history is the final filler, and a request without authorized Shopper context may return fewer than 20. An example request body is {"anchor_product_id":"item-42","userContext":{"city":"Montréal","region":"Québec","country":"Canada"},"quota":20}.

7.2 Error contract

HTTP status	Code	Condition
404	data_source_not_found	Submission names an unconfigured source
404	scope_not_found	Training or serving names an unconfigured property/catalog scope
409	training_run_active	A different idempotency key targets an active scope
404	training_run_not_found	Run ID does not exist
422	invalid_strategy	Global strategy used on anchored route, or anchored strategy used on global route
422	invalid_scope / invalid_product_id	Serving identifier violates the canonical storage bounds
422	FastAPI validation detail	Invalid enum, header, or limit
404	snapshot_not_found	No successful head exists for the scope
404	item_not_found	Head exists but has no set for the anchored product and strategy
503	snapshot_incomplete	Required global set is absent or persisted snapshot content fails domain validation
401	personalization_not_authorized	Interaction authentication or signed context is absent, invalid, or out of scope
409	interaction_payload_conflict / interaction_sequence_conflict	Interaction replay content differs or the next expected sequence is not contiguous
409	causal_not_observed	The requested acknowledged profile version was not observed within the bounded wait
422	personalization_capability_unavailable / invalid_causal_token	Scope/version/strategy is disabled, or a causal capability is invalid or expired
503	personalization_state_unavailable	A personalization write cannot commit or serving state is temporarily unavailable

A valid empty set returns 200, items as an empty list, and reason as insufficient_evidence. A stale snapshot also returns 200 with is_stale true, allowing downstream systems to keep operating after a newer training failure.

The complete authorization, privacy, profile, fallback, and causal-consistency contract is in shopper-personalization.md. The ordinary training and non-personalized serving endpoints remain trusted internal interfaces.

7.3 Training submission sequence

sequenceDiagram
    autonumber
    actor Caller
    participant API as FastAPI route
    participant Registry as DataSourceRegistry
    participant Runs as TrainingRunRepository
    participant DB as Control database

    Caller->>API: POST /v1/training-runs and Idempotency-Key
    API->>Registry: contains(data_source_id)
    alt source is unknown
        Registry-->>API: false
        API-->>Caller: 404 data_source_not_found
    else source is configured
        Registry-->>API: true
        API->>Runs: create(scope, key, requested_at)
        Runs->>DB: INSERT pending run
        alt insert succeeds
            DB-->>Runs: committed row
            Runs-->>API: new TrainingRun
            API-->>Caller: 202 pending run
        else idempotency unique conflict
            Runs->>DB: SELECT same scope and key
            DB-->>Runs: existing run
            Runs-->>API: existing TrainingRun
            API-->>Caller: 202 existing run
        else active-scope unique conflict
            Runs->>DB: SELECT pending or running run
            DB-->>Runs: active run
            Runs-->>API: TrainingRunActiveError
            API-->>Caller: 409 and active_run_id
        end
    end

7.4 Serving sequence

sequenceDiagram
    autonumber
    actor Consumer
    participant API as FastAPI route
    participant Snapshots as SnapshotRepository
    participant DB as Control database
    participant Domain as RecommendationSet validation

    Consumer->>API: GET recommendations with scope and limit
    API->>API: Validate route-strategy shape and CommerceScope
    API->>Snapshots: get_head_set(scope, strategy, anchor)
    Snapshots->>DB: Join serving_head to available snapshot
    alt no head
        DB-->>Snapshots: no row
        API-->>Consumer: 404 snapshot_not_found
    else head exists
        Snapshots->>DB: Read recommendation_set by snapshot, strategy, anchor_key
        alt set absent
            API-->>Consumer: 404 item_not_found or 503 snapshot_incomplete
        else set exists
            Snapshots->>Domain: Rehydrate and validate entries
            alt corrupt payload
                Domain-->>API: validation error
                API-->>Consumer: 503 snapshot_incomplete
            else valid payload
                API->>API: Slice first limit entries and compute staleness
                API-->>Consumer: 200 snapshot metadata, provenance, ranked items
            end
        end
    end

8. Source adapter and canonical stream contract

8.1 Source class diagram

classDiagram
    class SqlAlchemySourceAdapter {
        -Engine engine
        -CanonicalQueries queries
        -str isolation_level
        +open(scope, history_start, cutoff) SourceReadSession
    }

    class SourceReadSession {
        -Connection connection
        -Transaction transaction
        +str consistency_token
        +int max_partition_rows
        +stream_catalog(fetch_size) Iterator
        +stream_views(fetch_size) Iterator
        +stream_purchases(fetch_size) Iterator
        +stream_compatibility(fetch_size) Iterator
        +assert_consistency_token(token)
    }

    class CatalogRow {
        +str catalog_id
        +str product_id
        +bool is_eligible
        +str category_id
        +str brand
        +Decimal price
        +datetime created_at
        +str title
        +str description
    }

    class ViewRow {
        +str tracking_id
        +str catalog_id
        +str product_id
        +str session_id
        +datetime event_at
    }

    class PurchaseRow {
        +str tracking_id
        +str catalog_id
        +str product_id
        +str order_id
        +datetime event_at
        +Decimal quantity
        +str purchase_channel
        +str group_id
    }

    class CompatibilityRow {
        +str tracking_id
        +str catalog_id
        +str anchor_product_id
        +str product_id
    }

    SqlAlchemySourceAdapter --> SourceReadSession : creates
    SourceReadSession ..> CatalogRow : yields
    SourceReadSession ..> ViewRow : yields
    SourceReadSession ..> PurchaseRow : yields
    SourceReadSession ..> CompatibilityRow : yields

The adapter opens one transaction for catalog, view, purchase, and optional compatibility queries. Its default isolation is SERIALIZABLE for SQLite and REPEATABLE READ elsewhere, unless configuration overrides it. The transaction is always rolled back on exit because the source session is read-only from the service's perspective.

All configured SQL is executed with four bound parameters:

• tracking_id
• catalog_id
• history_start
• cutoff

Each configured query must filter and sort correctly; the adapter independently checks returned scope and monotonic ordering, plus the interaction streams' time windows, so a faulty query cannot silently mix data.

Stream	Required order	Additional validation
Catalog	product_id	Returned catalog_id equals scope
Views	session_id, event_at, product_id	Tracking/catalog match; history_start less than or equal to event_at less than cutoff
Each purchase channel	order_id, event_at, product_id	Same scope/window checks; quantity greater than zero
Compatibility	anchor_product_id, product_id	Tracking/catalog match; anchor and candidate differ

Rows are consumed through SQLAlchemy yield_per and mappings().partitions(fetch_size), with a default fetch size of 10,000. Query/driver failures become SourceIncompleteError. Invalid values, scope, order, or consistency become explicit source errors and fail the run. No all-row result, DataFrame, CSV, Parquet, COPY, UNLOAD, or raw staging path exists.

The consistency token is created with the read session and checked after each stream. In the current generic adapter it proves that pipeline stages are using the same session object; actual cross-query snapshot stability depends on the configured database isolation semantics and driver.

9. Training pipeline

9.1 Successful training sequence

sequenceDiagram
    autonumber
    participant Worker as TrainingWorker
    participant Runs as TrainingRunRepository
    participant Source as SourceReadSession
    participant Work as DerivedWorkStore
    participant Pipeline as generate_recommendations
    participant Publisher as SnapshotRepository
    participant DB as Control database

    Worker->>Runs: claim_next(worker_id, now)
    Runs->>DB: Lock expired run or oldest pending run
    DB-->>Worker: running run with cutoff and lease
    Worker->>Worker: Start heartbeat thread
    Worker->>Source: open(scope, cutoff minus 2 years, cutoff)
    Worker->>Pipeline: generate_recommendations(read, cutoff, budgets)
    Pipeline->>Source: stream catalog
    Pipeline->>Work: open selected aggregate engine
    Pipeline->>Source: stream views
    Pipeline->>Work: consume view groups into derived deltas
    Pipeline->>Source: stream online and optional offline purchases
    Pipeline->>Work: consume order groups into derived deltas
    Pipeline->>Work: select parameters, scores, truth, metrics, aggregates
    Pipeline->>Work: close and delete temporary directory
    Pipeline->>Pipeline: Build metadata similarity and registered strategy sets
    Pipeline-->>Worker: PipelineResult
    Worker->>Publisher: publish complete manifest and metrics
    Publisher->>DB: Insert BUILDING snapshot
    loop chunks of at most 1000 sets
        Publisher->>DB: Insert recommendation_set rows
    end
    Publisher->>DB: Lock build and run, verify lease, mark AVAILABLE, swap head, succeed run
    Publisher-->>Worker: snapshot_id
    Worker->>Publisher: retain newest five snapshots
    Worker->>Runs: retain run and diagnostic records
    Worker->>Worker: Stop and join heartbeat

9.2 Pipeline stages

1. Freeze the run cutoff at claim time and compute a two-year history start. Leap-day handling maps February 29 to February 28 when necessary.
2. Materialize the catalog, filter eligible products, and build product-to-category maps.
3. Split interaction groups temporally using a 28-day holdout. A whole session/order is assigned by its maximum event time, preventing one group from crossing train and holdout evidence.
4. Stream views and purchases into the configured temporary DuckDB or Polars work store containing only derived state.
5. Grid-search behavioral min-support and shrinkage values on train/holdout derived counts.
6. Load full-window co-view and co-purchase candidates using selected parameters.
7. Compute time-decayed view/purchase scores and recent momentum.
8. Build sparse metadata similarity in memory from eligible catalog rows.
9. Generate category-pair, compatibility-rule, and category/property popularity fallback candidates through typed providers.
10. Evaluate train-only provider chains and baselines, including warm, sparse, and strict-cold cohorts.
11. Rank and validate eight anchored sets for every eligible product and three global sets.
12. Return a complete eleven-strategy manifest, recommendation sets, and numeric metrics.

An empty eligible catalog still produces the three required global sets, each potentially with an insufficient_evidence outcome. When eligible products exist, publication requires all eleven strategy types to be observed across the payload.

Evidence assembly uses cohesive behavioral-search, popularity-search, interaction-limit, and work-resource values. A behavioral builder selects each signal's model and gathers its held-out truth and diagnostics; a popularity selector owns decay/momentum selection. A partition builder then constructs independent PartitionEvidence values: publication uses the full cutoff, while evaluation uses the training group partition and the exclusive holdout date. One provider factory builds both provider shapes from their respective evidence; evaluation never copies publication providers and replaces selected entries. Catalog metadata similarity remains deliberately shared.

9.3 Derived work-store ERD

erDiagram
    PAIR_DELTAS {
        varchar strategy
        varchar partition
        varchar left_product_id
        varchar right_product_id
        ubigint pair_count
    }
    SUPPORT_DELTAS {
        varchar strategy
        varchar partition
        varchar product_id
        ubigint support_count
    }
    POPULARITY_DELTAS {
        varchar signal
        varchar product_id
        decimal weight
    }
    DAILY_POPULARITY_DELTAS {
        varchar signal
        varchar product_id
        date event_day
        decimal weight
    }
    CATEGORY_PAIR_DELTAS {
        varchar partition
        varchar left_category_id
        varchar right_category_id
        ubigint pair_count
    }
    EXCLUSION_METRICS {
        varchar group_type
        ubigint group_count
        ubigint row_count
    }

These DuckDB tables intentionally have no raw tracking_id, session_id, order_id, or event_at columns. They contain mergeable deltas rather than canonical rows. In-memory counters flush when a key-count reaches batch_key_limit. DuckDB is configured with explicit memory, thread, spill-path, and maximum spill-size controls. Its TemporaryDirectory is deleted on normal exit and exceptions.

Each completed qualifying group becomes a sorted, distinct product/category membership summary with its temporal partition. Identical summaries are combined with an integer multiplicity. The source Session/Order identity is discarded; only a batch-local ordinal accompanies the summary. Within the DuckDB work store, counting algorithms live in pipeline/cooccurrence.py behind the CooccurrenceCountingStrategy protocol. DerivedWorkStore resolves the configured strategy once and owns buffering, backpressure, scheduling, and cleanup; it contains no backend-specific dispatch or counting implementation. DuckDBCooccurrenceStrategy preserves the native SQL/Arrow self-join path and remains the default. SparseMatrixCooccurrenceStrategy builds binary CSR incidence matrices per batch and temporal partition, then computes M.T @ (w * M) using unsigned 64-bit counts. Multiplicity weights are applied only once, so repeated groups are not squared. The diagonal gives product support; the strict upper triangle gives lexically oriented product/category pairs. AccelerateCooccurrenceStrategy uses Apple's cblas_dgemm on dense blocks of at most eight anonymous group summaries. It splits integer group multiplicities into 24-bit digits before each multiply, so every floating-point partial sum is an exactly representable integer; it reconstructs unsigned 64-bit counts before merging blocks. The product matrices are bounded by eight times the configured maximum group membership count on either axis. This backend requires a native macOS worker: selecting it on Linux raises a clear runtime error when the work store is created. The TorchCooccurrenceStrategy uses float32 MPS products with 16-bit weight digits; MLXCooccurrenceStrategy uses GPU products with eight-bit digits, which remain exact even under MLX's reduced-precision float32 matmul mode. ANECooccurrenceStrategy uses fp16 ANEForge products with eight-bit digits. In each case, at most eight digit terms contribute to a cell, and unsigned 64-bit counts are reconstructed on the host. Accelerator output must contain finite integers; otherwise the run fails. The ANE backend caches at most 16 compiled block shapes per work store, serializes calls to its model, and releases programs at work-store cleanup. These backends are opt-in and require native Apple Silicon with their respective optional dependencies; they do not silently fall back to CPU. ANEForge uses private Apple interfaces and may break after an OS update. The current Linux Docker worker cannot use Apple's frameworks. None of the backends filters by minimum support or top-N before all batches have been merged. Only the existing derived aggregate tables are persisted. Raw rows, timestamps, source identities, and source credentials never enter background tasks.

Set "cooccurrence_backend" to "sparse", "accelerate", "torch", "mlx", or "ane" inside the deployment JSON's pipeline object to opt in; use "duckdb" or omit the field to retain the original path. Install the matching extra (uv sync --extra torch, --extra mlx, or --extra ane) on the worker before selecting an optional backend. Invalid values fail configuration loading. The choice applies to Also Viewed and Frequently Bought Together (including category pairs), not metadata similarity. It changes the configuration fingerprint, not scoring or ranking.

Summary batches are bounded by their possible expanded contributions: n*(n+1)/2 for product support plus pairs, and c*(c-1)/2 for category pairs, counted once per distinct summary. A completed group can exceed batch_key_limit by at most that group's bounded contribution. Popularity and exclusion accumulation retain the streaming consumer's existing semantics. Arrow preserves UTF-8 identifiers (including NUL), integer counts, and dates without per-value string conversion. Decimal weights still use exact text and DuckDB's existing DECIMAL(38,8) cast/rounding boundary.

With pipeline.threads > 1, at most floor(threads / 2) background native tasks use separate DuckDB cursors while one source consumer continues streaming. Submission waits before allocating another batch when all slots are occupied. During ingestion DuckDB receives threads - 1 CPUs, with floor(threads / 2) reserved for calling threads and the remainder for its shared native pool. This reserves one CPU for streaming and avoids multiplying native pools by task count. Reads restore the full configured DuckDB query budget after joining tasks. threads: 1 is synchronous. The input summary/counter buffers and in-flight Arrow batches are bounded; increasing threads also increases their possible memory footprint. DuckDB's memory and spill limits remain shared across all cursors. Arrow/Python input buffers and SciPy or accelerator working matrices are outside the DuckDB memory limit and must be included in worker sizing. The sparse backend never allocates a dense Catalog-by-Catalog matrix: each sparse product has at most twice the batch's possible contribution count in stored entries (including the bounded group overshoot). Sparse counting uses the same bounded concurrent batch tasks, not a separately configured native thread pool; selecting sparse matrices does not itself guarantee higher CPU utilization.

Reads and stream completion join outstanding tasks and propagate failures. Error cleanup joins or cancels every task before closing DuckDB and deleting scratch. Source validation and group boundaries remain on one ordered stream and its original consistent transaction. Metadata similarity also uses pipeline.threads for native sparse multiplication; the container enables OpenMP. See the parallel reduction plan for decisions, verification, and performance evidence. This does not promise full CPU utilization during source I/O, Python validation/popularity reduction, or small-input processing.

9.3.1 Polars aggregation engine

pipeline.aggregation_backend selects duckdb (default) or polars for the complete derived work store. pipeline/workstore_factory.py assembles the selection; pipeline/ingestion.py shares the canonical streaming reducer, privacy boundary, group limits, geography, and temporal partition rules. pipeline/polars_workstore.py implements the same work-store queries using Polars expressions, joins, aggregations, and deterministic ranking. It opens no DuckDB connection. Metadata similarity and downstream snapshot publication/serving are shared.

The Polars implementation writes only the six derived table schemas above to temporary Parquet row groups, with exact DECIMAL(38,8) quantities rounded half-up at the same batch boundary. Anonymous membership batches stay transient; Polars explodes their distinct members and self-joins within each batch-local ordinal. UInt64 deltas are summed in Int128 across batches to avoid UInt64 wraparound. Support thresholds and category constraints precede top-N ranking only after all selected batches have been combined. Floating scores may differ at machine precision between engines; count, partition, eligibility, and lexical tie-breaking contracts are identical.

Ingestion is synchronous with one bounded native batch at a time. Writers rotate after at most 128 batch writes; each table retains at most 16 files before compaction. Scratch is removed on success and exceptions. The disk budget is checked after each batch write, footer close, and compaction; transient overshoot can include one write or a compaction copy. The memory budget checks materialized batch/result sizes, not peak process memory or native query intermediates. Lazy reads use Polars' streaming engine; its own internal spill is outside the owned Parquet footprint. Use worker/container limits for a hard process memory or total filesystem bound. This implementation has correctness parity coverage, not a scale or performance qualification.

Polars uses its process-wide native thread pool. Set POLARS_MAX_THREADS before starting the worker to cap that pool; pipeline.threads still controls shared metadata similarity and DuckDB, but cannot resize an initialized Polars pool. This follows the Polars thread-pool contract. The existing cooccurrence_backend field selects counting accelerators only within the DuckDB work store. Leave it omitted/default when selecting Polars; a non-default override with Polars is rejected rather than ignored. The aggregation selection participates in the configuration fingerprint, preventing drift comparisons across different engine configurations.

9.4 Group reduction and cardinality controls

Views group by session_id. Purchases group by purchase_channel plus order_id. Each group uses a set of distinct products for support and pair generation, so repeated lines/views do not inflate pair support. Popularity keeps every view and the full purchase quantity.

The default inclusive distinct-product limits are configurable through PipelineConfig:

Group	Maximum distinct products	Maximum emitted pairs at limit
Browsing session (view_session_distinct_product_limit)	100	4,950
Purchase order (purchase_order_distinct_product_limit)	200	19,900

If a group exceeds its limit, the entire group contributes no pair, support, or category-pair evidence. Its rows still contribute to popularity and daily popularity. Excluded group and row counts are recorded in run metrics. This avoids partial-pair bias and bounds per-group memory.

GroupAccumulator owns current-context membership, maximum event time, geography consistency, and oversized state. It clears membership immediately when a group exceeds its bound and emits an identity-free GroupSummary when the context closes. Typed view and purchase adapters retain their different grouping/weighting semantics. A separate derived buffer owns pair, geographic, popularity, daily, and exclusion flush thresholds. Distinct exclusion-count keys are also flushed at the configured batch-key limit. Existing native backpressure and scratch cleanup remain owned by DerivedWorkStore.

10. Recommendation algorithms

10.1 Behavioral co-occurrence

For products i and j:

cosine(i,j) = pair_support(i,j) / sqrt(support(i) * support(j))

score(i,j) = cosine(i,j) * pair_support(i,j) / (pair_support(i,j) + shrinkage)

pair_support is the number of distinct qualifying groups containing both products; support is the number of groups containing each product. The default minimum support is 2 and default shrinkage is 10. During evaluation the pipeline tests min-support values 2, 3, and 5 and shrinkage values 5, 10, and 20. It selects highest NDCG@20, then Recall@20, catalog coverage, and finally lower min-support and shrinkage as stable tie breakers. If the holdout has no anchors, the defaults are retained.

The selected aggregation engine computes scores, expands each undirected pair into two anchored directions, and keeps at most 200 generation candidates per anchor before downstream filtering.

10.2 Metadata similarity

Only eligible products with at least one metadata value participate. Feature blocks are:

• categorical one-hot features for category, brand, and price bucket;
• word TF-IDF over combined title and description with one- and two-word n-grams;
• character-within-word TF-IDF with 3- through 5-character n-grams, weighted by 0.5.

Price buckets combine within-category quartiles with a coarse logarithmic price bucket. The sparse combined matrix is L2-normalized. sparse-dot-topn multiplies the matrix by its transpose and retains at most 201 raw similarities so the self-match can be removed while keeping up to 200 candidates. Non-positive similarities are discarded; remaining ties use ascending Product ID.

10.3 Popularity

Best Sellers uses purchase quantity and Most Viewed counts views. Their half-lives are selected by chronological holdout quality from the configured grids, with 30 and 7 days as the respective fallbacks when no holdout evidence exists. Daily-bucket decay is computed as:

contribution = weight * exp(-ln(2) * age_days / half_life_days)

The work store uses midnight daily buckets, which intentionally trades sub-day precision for a bounded aggregate representation.

10.4 Trending

Trending combines view count and purchase quantity with configurable non-negative weights. Its recent window, baseline window, and alpha are selected from configured grids by chronological holdout quality, with 7 days, 28 days, and 0.1 as the no-evidence fallbacks:

momentum = log2((recent_count / 7 + alpha) / (baseline_count / 28 + alpha))

score = momentum * log1p(recent_count)

Only products with recent evidence and positive momentum are eligible.

10.5 Fallback ranking

flowchart LR
    subgraph FBT[Frequently Bought Together]
        f1[Co-purchase] --> f2[Category-pair purchase]
        f2 --> f3[Compatibility rule]
        f3 --> f4[Category recent purchases]
        f4 --> f5[Property recent purchases]
    end

    subgraph AV[Also Viewed]
        a1[Co-view] --> a2[Metadata similarity]
        a2 --> a3[Category recent views]
        a3 --> a4[Property recent views]
    end

    subgraph SI[Similar Items]
        s1[Metadata similarity only]
    end

One immutable strategy registry owns anchored/global shape, metric identity, and allowed evidence tiers. A factory orders CandidateProvider implementations into the displayed chain. The ranker drains each provider, filters the anchor, ineligible products, and previously used IDs, then stops at 100. Scores are compared only inside the same evidence tier. Provenance source_tier and evidence_type identify the winning tier; fallback_reason is fallback for any non-primary tier.

11. Evaluation and training metrics

The final 28 days form the holdout. A group belongs to holdout when its maximum event timestamp is on or after the holdout boundary; otherwise it belongs to train. Recall and NDCG at 10, 20, and 100 are computed per anchor and averaged. Support cohorts are:

Cohort	Train support
strict_cold	0
sparse	1 through 3
warm	Greater than 3

Evaluation builds the same provider chains from train-only interaction evidence, then measures the final tier-filled rankings against holdout truth. It covers selected co-view/co-purchase models, raw-pair and popularity baselines, metadata Similar Items, the three global strategies, and category strategies with relation-restricted holdout truth. Catalog coverage, top-one-percent exposure concentration, fallback rate, and warm/sparse/strict-cold results are retained. All metrics are numeric so they can be stored directly in training_run.metrics.

Core operational metrics include:

• catalog and eligible product counts;
• streamed view and purchase row counts;
• unique derived view and purchase pair counts;
• temporary work-store bytes;
• excluded session/order and row counts;
• selected behavioral parameters;
• per-strategy and per-cohort Recall/NDCG at 10, 20, and 100 and anchor count;
• catalog coverage, popularity concentration, fallback rate, and raw-pair/popularity baselines;
• total duration in seconds.

The focused test suite verifies metric semantics. The 200,000-product / 100-million-view / 100-million-purchase capacity and latency objectives require scripts/qualify.py and a provisioned qualification environment; local tests do not establish them.

When quality_drift.enabled is true, the worker compares the just-published run with a bounded history of successful runs for the same Commerce Scope whose model, configuration, and Training Evidence Schema versions match. Median/MAD robust scores and configured absolute/relative floors produce within_expected_range, warning, critical, or an explicit insufficient/incomparable status. Assessment is best-effort after publication: failure or threshold crossing cannot roll back the snapshot or change the serving head. The schema version is independent of the OpenTelemetry signal schema, and the report is aggregate operational evidence—not a business-outcome claim.

12. Durable persistence

12.1 Service-owned ERD

erDiagram
    TRAINING_RUN {
        varchar run_id PK
        varchar data_source_id
        varchar tracking_id
        varchar catalog_id
        varchar idempotency_key
        varchar config_version
        varchar model_version
        varchar status
        timestamptz requested_at
        timestamptz started_at
        timestamptz finished_at
        timestamptz data_cutoff
        varchar snapshot_id
        varchar failure_category
        json metrics
        varchar metrics_schema_version
        varchar lease_owner
        timestamptz lease_expires_at
        int recovery_count
    }

    RUN_FAILURE_DIAGNOSTIC {
        varchar diagnostic_id PK
        varchar run_id FK
        timestamptz observed_at
        varchar failure_category
        varchar diagnostic_schema_version
    }

    RECOMMENDATION_SNAPSHOT {
        varchar snapshot_id PK
        varchar run_id UK
        varchar data_source_id
        varchar tracking_id
        varchar catalog_id
        varchar state
        timestamptz generated_at
        timestamptz data_cutoff
        int freshness_hours
        varchar config_version
        varchar model_version
        json strategy_manifest
    }

    RECOMMENDATION_SET {
        varchar snapshot_id PK, FK
        varchar strategy PK
        varchar anchor_key PK
        varchar geo_level PK
        varchar geo_key PK
        varchar outcome
        json entries
    }

    SERVING_HEAD {
        varchar data_source_id PK
        varchar tracking_id PK
        varchar catalog_id PK
        varchar snapshot_id FK
    }

    RECOMMENDATION_CANDIDATE_FEATURE {
        varchar snapshot_id PK, FK
        varchar product_id PK
        varchar category_id
        varchar brand
        boolean is_eligible
    }

    RECOMMENDATION_ANN_ARTIFACT {
        varchar snapshot_id PK, FK
        varchar representation_version
        int dimensions
        json product_ids
        json eligible_ids
        bytes vectors
        bytes index_bytes
        bytes input_vectors
        bytes query_weights
        varchar digest
    }

    SHOPPER_INTERACTION {
        varchar data_source_id PK
        varchar property_id PK
        varchar catalog_id PK
        varchar shopper_id PK
        int sequence PK
        varchar interaction_id
        json payload
        varchar payload_hash
        int profile_version
        timestamptz retention_expires_at
    }

    SHOPPER_INTERACTION_IDEMPOTENCY {
        varchar data_source_id PK
        varchar property_id PK
        varchar interaction_id PK
        varchar catalog_id
        varchar shopper_id
        varchar payload_hash
        int committed_sequence
        int profile_version
        timestamptz expires_at
    }

    SHOPPER_PROFILE {
        varchar data_source_id PK
        varchar property_id PK
        varchar catalog_id PK
        varchar shopper_id PK
        int profile_version
        int source_watermark
        json bounded_affinities
        varchar lifecycle_state
        timestamptz retention_boundary
    }

    SHOPPER_SUPPRESSION {
        varchar data_source_id PK
        varchar property_id PK
        varchar catalog_id PK
        varchar shopper_id PK
        varchar reason
        int barrier_watermark
        int profile_version
        timestamptz expires_at
    }

    SHOPPER_DELETION_RECEIPT {
        varchar receipt_id PK
        varchar data_source_id
        varchar property_id
        varchar catalog_id
        varchar shopper_id
        int requested_sequence
        varchar status
    }

    TRAINING_RUN ||--o{ RUN_FAILURE_DIAGNOSTIC : records
    TRAINING_RUN ||..o| RECOMMENDATION_SNAPSHOT : logical_run_id
    RECOMMENDATION_SNAPSHOT ||--o{ RECOMMENDATION_SET : contains
    RECOMMENDATION_SNAPSHOT ||--o{ SERVING_HEAD : selected_by
    RECOMMENDATION_SNAPSHOT ||--o{ RECOMMENDATION_CANDIDATE_FEATURE : describes
    RECOMMENDATION_SNAPSHOT ||--o| RECOMMENDATION_ANN_ARTIFACT : retrieves_with

The run-to-snapshot association is logical and unique on recommendation_snapshot.run_id; the current migration does not declare it as a foreign key. training_run.snapshot_id is likewise a denormalized outcome reference without a database foreign key. The enforced foreign keys are:

• run_failure_diagnostic.run_id to training_run.run_id with cascade delete;
• recommendation_set.snapshot_id to recommendation_snapshot.snapshot_id with cascade delete;
• serving_head.snapshot_id to recommendation_snapshot.snapshot_id;
• recommendation_candidate_feature.snapshot_id to recommendation_snapshot.snapshot_id with cascade delete;
• recommendation_ann_artifact.snapshot_id to recommendation_snapshot.snapshot_id with cascade delete.

The ERD summarizes durable ownership; SQLAlchemy metadata and Alembic migrations define exact column types/defaults. Later migrations add typed recent profiles, geographic set keys, and ANN artifacts.

Shopper tables deliberately repeat the full property/Catalog/Shopper key instead of depending on a global shopper identity. The state repository updates the interaction ledger, idempotency record, profile projection, and lifecycle barrier transactionally. Suppression and deletion receipts prevent late writes or retries from resurrecting state after opt-out or deletion.

12.2 Indexes and constraints

Name	Purpose
uq_training_run_idempotency	One row per scope and idempotency key
uq_training_run_active_scope	Partial unique index allowing only one pending/running row per scope
ix_training_run_claim	Ordered pending-run lookup by status, requested time, and ID
ix_training_run_lease	Expired running-run lookup
ix_recommendation_snapshot_scope	Snapshot listing/retention by scope, state, and generation time
recommendation_set composite PK	One set per snapshot, strategy, anchor_key, geo_level, and geo_key
recommendation_ann_artifact PK	At most one immutable retrieval artifact per snapshot
serving_head composite PK	Exactly one current head per scope
uq_shopper_interaction_source_property_id	One meaning for an interaction ID within its data source and Commerce Property
shopper_profile composite PK	One bounded projection per full property/Catalog/Shopper key
uq_shopper_deletion_receipt_request	Idempotent deletion receipt per Shopper sequence request
recommendation_candidate_feature composite PK	One metadata companion row per snapshot and Product ID

Global sets encode anchor_key as the empty string. Anchored sets store the Product ID.

12.3 Stored recommendation payload

Each recommendation_set row holds at most 100 pre-ranked entries in JSON/JSONB. An entry contains:

Field	Meaning
product_id	Candidate inside the snapshot scope
rank	Contiguous one-based order
score	Strategy/tier-specific numeric score
confidence_tier	high, medium, or low
provenance.source_tier	The tier that supplied the retained candidate
provenance.evidence_type	Typed evidence classification
provenance.support	Pair/category support when applicable
provenance.fallback_reason	Null for primary evidence; fallback for later tiers

The repository rehydrates this JSON into domain objects on every read. Domain validation turns malformed ranks, duplicates, self-recommendations, impossible anchors, invalid enum values, or non-numeric fields into a 503 instead of leaking corrupt data.

13. Run lifecycle, leasing, and atomic publication

13.1 State machine

stateDiagram-v2
    [*] --> Pending: accepted
    Pending --> Pending: same idempotency key returns existing row
    Pending --> Running: worker claims and fixes cutoff
    Running --> Running: lease heartbeat or one recovery
    Running --> Succeeded: complete snapshot atomically published
    Running --> Failed: classified source, resource, or computation failure
    Running --> Failed: second expired lease becomes worker_lost
    Succeeded --> [*]
    Failed --> [*]

    state SnapshotPublication {
        [*] --> Building
        Building --> Available: validate and swap head
        Building --> Discarded: any publication exception
    }

13.2 Claim and recovery sequence

sequenceDiagram
    autonumber
    participant Worker as Worker instance
    participant Runs as TrainingRunRepository
    participant DB as Control database

    Worker->>Runs: claim_next(now, five-minute lease)
    Runs->>DB: SELECT expired RUNNING FOR UPDATE SKIP LOCKED
    alt expired run exists and recovery_count is zero
        Runs->>DB: Set new owner/expiry and recovery_count to one
        DB-->>Worker: recovered RUNNING run
    else expired run exists and was already recovered
        Runs->>DB: Mark FAILED worker_lost and clear lease
        DB-->>Worker: no work this tick
    else no expired run
        Runs->>DB: SELECT oldest PENDING FOR UPDATE SKIP LOCKED
        alt pending run exists
            Runs->>DB: Mark RUNNING, set start/cutoff/owner/expiry
            DB-->>Worker: claimed run
        else queue empty
            DB-->>Worker: no work
        end
    end

The default database lease is five minutes. While training, a daemon thread renews it every 60 seconds by default. Renewal requires both running status and matching lease owner. If renewal fails, the worker refuses to publish. A recovered run reuses its original data_cutoff, ensuring the same history boundary, and the publisher deletes an abandoned building snapshot for that run before restarting publication.

13.3 Atomic publication

Publication separates chunked set/feature insertion from the visibility transaction. When ANN is enabled, the artifact is inserted in that final transaction too, so its size contributes to activation cost:

1. Validate that completed_strategies equals all registered snapshot strategy enum values.
2. Delete an abandoned building snapshot for the same run.
3. Insert a new recommendation_snapshot in building state.
4. Serialize and insert recommendation sets and candidate feature companions in batches of 1,000 by default. Validate an optional ANN artifact before creating building state.
5. For a worker publication, verify required strategy presence, running run state, and lease owner.
6. In one transaction, insert the optional ANN artifact, mark the snapshot available, replace the scope's serving_head, mark the run succeeded, persist metrics, and clear the lease.
7. If any step after build creation fails, discard its unpublished snapshot and staged companions.

Readers join through serving_head and require state available, so partially inserted building rows are invisible. A failed new run never modifies the previous head.

SnapshotRepository.publish coordinates separate helpers for building-snapshot creation, bounded set staging, publication ownership checks, head activation, and run completion. The last three receive the same connection from the coordinator's transaction; they do not commit independently. Failure tests cover both an interrupted input stream after a committed staging batch and a failed run-completion update after head activation, asserting that staging is removed and the prior head and running run are preserved.

14. Failure handling

14.1 Worker failure mapping

Exception boundary	Stored failure_category	Effect
SourceOrderError, SourceScopeError, SourceValidationError	source_invalid	Run fails; current head preserved
SourceConsistencyError, SourceIncompleteError	source_incomplete	Run fails; current head preserved
duckdb.OutOfMemoryException	resource_exhausted	Run fails; scratch closes; current head preserved
Missing source adapter	configuration_invalid	Run fails before reading source
Unclassified exception	strategy_computation_error	Run fails; current head preserved
LeaseLostError or TrainingRunLeaseLostError	No write by stale worker	Current owner/recovery path decides outcome
Lease expires twice	worker_lost	Repository marks terminal failure

mark_failed updates only a running row and may additionally require the expected worker owner. It clears the lease and inserts one bounded run_failure_diagnostic row. The worker suppresses a late failure update if another actor already changed the run, avoiding ownership races.

14.2 Failure-preserves-head sequence

sequenceDiagram
    autonumber
    participant Worker
    participant Source
    participant Work as Temporary DuckDB
    participant Runs as TrainingRunRepository
    participant DB as Control database
    participant API

    Worker->>Source: Stream canonical rows
    Source--xWorker: Validation, ordering, consistency, or driver error
    Worker->>Work: Exit context and delete scratch
    Worker->>Runs: mark_failed(category, expected owner)
    Runs->>DB: Update run and insert diagnostic
    Note over DB: serving_head is unchanged
    API->>DB: Read existing serving_head
    DB-->>API: Previous available snapshot

15. Retention and freshness

Current code defaults are:

Artifact	Retention/freshness	Enforcement
Successful snapshots	Newest five per scope, always including head	After successful publication
Training-run metadata	90 days after terminal completion	After successful publication
Failure diagnostics	30 days	After successful publication
Snapshot freshness	48 hours from generated_at	Written by worker and evaluated at serve time

Deployment JSON may override all three retention values. Freshness has a deployment default and optional exact (data_source_id, tracking_id, catalog_id) overrides; the worker resolves and stores the selected value during publication.

Expired snapshots are not automatically unavailable. Freshness is response metadata, and stale heads continue serving until a successful run swaps the pointer. Retention never deletes the head.

16. Security, privacy, and isolation

Implemented controls:

• API requests name only configured data_source_id values; connection URLs and SQL are deployment configuration, not request data.
• Source SQL parameters are bound through SQLAlchemy text execution.
• Returned catalog/tracking/time scope is validated even when configured SQL is faulty.
• CommerceScope is included in run, snapshot, head, and lookup keys.
• Raw rows from the source Interaction Dataset are streamed and never stored in the control database or DuckDB. This is distinct from authorized, retention-bounded personalization commands committed through the Interaction API.
• Failure diagnostics contain category and time, not raw event content.
• Corrupt persisted recommendation payloads fail closed with 503.
• Resource use is bounded by fetch partitions, group limits, derived-delta flushes, top-N limits, DuckDB memory, threads, and spill capacity.
• Personalization binds bearer-derived property principals, signed Shopper context, and causal tokens to one data source, Commerce Property, Catalog, and opaque Shopper ID.
• Personalization logs, telemetry, responses, and simulation evidence omit raw Shopper identifiers, credentials, profile contents, and authorization assertions.
• Observability configuration resolves secrets only at runtime and restricts metric dimensions to bounded vocabularies; raw URLs, request bodies, and recommendation payloads are excluded.

Important trust boundary: the ordinary training and non-personalized serving APIs deliberately have no application authentication or authorization. Network, gateway, and database permissions must restrict those internal APIs and data-source credentials. The additive personalization write path does authenticate before parsing its request body; personalized reads degrade to an explicit safe fallback when authorization is absent or invalid. Configured SQL is trusted operator input; it is not user input, but it must still be reviewed and certified per source dialect.

17. Operations

17.1 Startup order

1. Install the package with the PostgreSQL extra and any required source-dialect extras.
2. Provision the service PostgreSQL database and least-privilege source credentials.
3. Copy and edit config/data_sources.example.json; expose its path and URL environment variables.
4. Run alembic upgrade head.
5. Start recommendations-api and smoke-test submission/status/serving error behavior.
6. Start one or more recommendations-worker processes.
7. Submit a small scope and inspect its terminal metrics before production-scale runs.

17.2 Scaling model

• Scale API replicas horizontally; they are stateless apart from PostgreSQL.
• Scale workers across independent scopes. One-active-scope constraints prevent concurrent training of the same data source, property, and catalog.
• A single run is executed by one worker and its configured local derived work store; the pipeline is not distributed.
• Increase worker CPU/RAM/scratch through PipelineConfig only after measuring. Candidate and group semantics are independent of flush size and DuckDB thread count.

17.3 Rollback and recovery

Stop new worker claims before rolling back worker code. API code may continue serving the current available snapshot. A failed build is invisible and discardable. Downgrade the migration only when no deployed process needs the schema and retained payloads remain compatible. Source databases are never migrated by this service.

17.4 Observability boundary

The service persists versioned per-run aggregate evidence and bounded failure categories, visible through GET /v1/training-runs/{run_id}. Separately, the optional process-owned observability runtime emits traces, metrics, and structured logs to one OTLP Collector endpoint. It is disabled by default and fails open with respect to recommendation behavior. Recommendation-domain signal names, dimensions, dashboards, alerts, and response procedures remain owned by this repository; generic SDK/export/runtime safety remains in the sibling telemetry package. The API has no application metrics endpoint or health route; Compose health for the recommendation API uses a non-mutating OpenAPI request, while the Simulation Storefront exposes /health and /ready.

See recommendations-operations.md and ../operations/recommendations.toml. Capacity and latency qualification still uses retained evidence from scripts/qualify.py; exported telemetry alone is not a Qualification Claim.

18. Verification

18.1 Test architecture

Suite	What it proves
tests/unit	Pipeline algorithms, quality/drift, personalization, admission control, simulation policy/lifecycle/evidence, observability, and DuckDB aggregate behavior
tests/contract	HTTP and observability adapter contracts, bounded source semantics, and absence of raw staging paths
tests/integration	Full training/serving lifecycle, migrations, PostgreSQL behavior, personalization storage/candidates, and storefront-to-runner flows
tests/delivery	Repository command surface, CI workflow, packaging assumptions, and installed-artifact verification support

SQLite and fixed clocks make focused tests fast and deterministic. PostgreSQL-specific transaction, row-locking, and partial-index behavior still requires a disposable PostgreSQL environment for full production-equivalent verification.

18.2 Required local gates

make setup
make doctor
make test
make smoke

Use make test-focused TEST=tests/path.py::test_name during development and make verify for the complete static, layered-test, build, and installed-artifact gate. Use make migration-check for migration changes and make compose-check for Docker or Compose changes. PostgreSQL-marked tests require an explicitly disposable TEST_CONTROL_DATABASE_URL; the supported ladder and CI contract are in testing.md.

Export and inspect the HTTP schema with:

uv run --frozen python scripts/export_openapi.py artifacts/openapi.json

Capacity and latency qualification uses scripts/qualify.py as documented in README.md. Monthly availability cannot be inferred from repository tests; it requires production request telemetry.

19. Module reference

Module	Responsibility	Main public surface
recommendations.domain	Framework-independent values, strategy registry, enums, invariants, normalization	StrategyDefinition, CommerceScope, TrainingRun, RecommendationSet
recommendations.contracts	HTTP request/response serialization	TrainingRunRequest/Response, RecommendationResponse, ErrorResponse
recommendations.config	Strict environment and JSON deployment parsing	DeploymentConfig, PersonalizationConfig, FreshnessConfig, RetentionConfig, DataSourceConfig
recommendations.api	HTTP composition and route/error mapping	create_app, main
recommendations.serving	Snapshot-only application decisions and candidate-loading strategies	RecommendationServingService, SnapshotReader, ServingFailure
recommendations.geography	Source-local hierarchical normalization and keys	GeoKey, GeoLevel
recommendations.swimlane	Ordered quota fill, filters, and recent-seed fusion	SwimlanePlan, compose_swimlane, recent_seed_candidates
recommendations.swimlane_service	Compile contextual lookups and resolve against one head	resolve_swimlane
recommendations.ann	Versioned artifact, validation, residency, native retrieval, exported query inference	AnnArtifact, LoadedAnnIndex, AnnSnapshotRetriever
recommendations.two_tower_torch	Optional aggregate-pair model training and export	TwoTowerArtifactBuilder
recommendations.source	Source protocols, canonical typed rows, and bounded relational streams	SourceAdapter, SourceRead, SqlAlchemySourceAdapter
recommendations.pipeline.workstore	Ephemeral derived aggregation and SQL scoring	DerivedWorkStore
recommendations.pipeline.polars_workstore	Alternative complete derived aggregate engine	PolarsWorkStore
recommendations.pipeline.workstore_factory	One-time aggregate engine assembly	create_work_store
recommendations.pipeline.ingestion	Shared streaming reduction and identity-free derived buffers	DerivedGroupConsumer
recommendations.pipeline.groups	Bounded transient group state and typed interaction adaptation	GroupAccumulator, GroupSummary, InteractionAdapter
recommendations.pipeline.cooccurrence	Pure in-memory reducer/reference scoring functions	reduce_view_groups, reduce_purchase_groups, regularized_cosine_score
recommendations.pipeline.content	Sparse metadata cosine	build_similar_items
recommendations.pipeline.compatibility	Bounded published and train-only compatibility-rule candidates	CompatibilityCandidateSets, build_compatibility_candidate_sets
recommendations.pipeline.popularity	Pure decay and momentum reference functions	decayed_view_scores, decayed_purchase_scores, trending_scores
recommendations.pipeline.evaluation	Temporal partitions, Recall, NDCG, cohorts	assign_group_partitions, evaluate_rankings
recommendations.pipeline.fallbacks	Candidate-provider chain, eligibility, deduplication, confidence, provenance	CandidateProvider, create_candidate_chain, rank_candidate_chain
recommendations.pipeline.run	Full batch orchestration and metrics	generate_recommendations, PipelineResult
recommendations.storage	SQLAlchemy schema, repositories, publication, retention	TrainingRunRepository, SnapshotRepository, ServiceStore
recommendations.quality	Version-aware, bounded quality and drift assessment	QualityDriftConfig, QualityHistory, assess_quality_and_drift
recommendations.personalization.*	Authorization, ordered interaction commits, lifecycle state, profiles, causal tokens, and reranking	ShopperInteractionService, PersonalizedRecommendationService, SqlAlchemyShopperStateRepository
recommendations.personalization_evaluation	Leakage-resistant offline comparison and experiment contracts	evaluation and assignment value objects/functions
recommendations.observability.*	Domain-owned signal schema and adapter into the sibling runtime	Observability, FailSafeObservability, ObservabilityRuntime
recommendations.synthetic.*	Deterministic generated relational source and lineage artifacts	profiles, generator, shared MaterializedSource, SQLite/PostgreSQL materializers, manifests, CLI
recommendations.simulation.*	Local/test storefront, source, scenarios, visitor execution, checkpoints, lifecycle, and evidence	SimulationHarnessService, SyntheticVisitorRunner, create_storefront_app
recommendations.worker	Lease-owned execution and exception classification	TrainingWorker, main
serving_limits.core/config/asgi	Reusable, application-independent token-bucket/concurrency admission foundation	AdmissionController, LimitPolicy, InMemoryLimitState, AdmissionControlMiddleware

19.1 Runtime collaboration class diagram

classDiagram
    class TrainingWorker {
        -str worker_id
        -ServiceStore store
        -dict source_adapters
        -Clock clock
        -float heartbeat_seconds
        -PipelineConfig pipeline_config
        +run_once() bool
    }

    class ServiceStore {
        +Engine engine
        +TrainingRunRepository training_runs
        +SnapshotRepository snapshots
        +SqlAlchemyShopperStateRepository shopper_state
    }

    class TrainingRunRepository {
        +create(scope, key, requested_at) TrainingRun
        +get(run_id) TrainingRun
        +claim_next(worker_id, claimed_at) TrainingRun
        +renew_lease(run_id, worker_id, renewed_at) bool
        +mark_failed(run_id, category)
        +retain_operational_records(now)
    }

    class SnapshotRepository {
        +publish(scope, run_id, sets, manifest) str
        +get_head_set(scope, strategy, anchor) SnapshotLookup
        +retain(scope, keep)
    }

    class SourceAdapter {
        <<Protocol>>
        +open(scope, history_start, cutoff) SourceRead
    }

    class SqlAlchemySourceAdapter {
        +open(scope, history_start, cutoff) SourceReadSession
    }

    class DerivedWorkStore {
        +consume_views(rows)
        +consume_purchases(rows)
        +load_behavioral_candidates() dict
        +load_decayed_scores() dict
        +load_trending_scores() dict
    }

    class PipelineResult {
        +tuple recommendation_sets
        +frozenset completed_strategies
        +dict metrics
    }

    TrainingWorker --> ServiceStore
    ServiceStore *-- TrainingRunRepository
    ServiceStore *-- SnapshotRepository
    ServiceStore *-- SqlAlchemyShopperStateRepository
    TrainingWorker --> "1..*" SourceAdapter
    TrainingWorker ..> PipelineResult : receives
    SourceAdapter <|.. SqlAlchemySourceAdapter
    SqlAlchemySourceAdapter --> SourceReadSession
    PipelineResult ..> DerivedWorkStore : built using

20. Implementation guarantees and qualification limits

The following are directly enforced by code and focused tests:

• asynchronous durable submission and idempotency;
• one active run per scope;
• bounded ordered canonical streaming with scope/time validation;
• no raw interaction staging;
• deterministic eleven-strategy snapshot generation;
• whole-group temporal partitioning and oversized-group exclusion;
• provenance-bearing, tier-ordered, pre-ranked results;
• leased worker recovery and stale-worker publication prevention;
• invisible building snapshots and atomic head replacement;
• stale-head preservation after failure;
• bounded snapshot/run/diagnostic retention;
• fail-closed serving of corrupt stored results;
• default-off, exact-scope personalization capability negotiation;
• authenticated, ordered, idempotent personalization writes and bounded Snapshot-only reranking;
• immediate opt-out/deletion barriers plus bounded worker reconciliation;
• strict local/test simulation target policy, deterministic scenario identity, and bounded evidence;
• fail-open optional telemetry with bounded dimensions and secret-safe configuration;
• reusable in-process admission semantics without a distributed-capacity claim.

The following require environment-specific evidence and must not be inferred from unit tests:

• PostgreSQL behavior under production concurrency and failover;
• each non-SQLite source dialect's repeatable-read/streaming certification;
• completion within 12 hours at the approved 200k/100M/100M scale;
• serving p95/p99 and training API p95 latency objectives;
• monthly availability objectives;
• production privacy/legal readiness or controlled-experiment evidence for personalization;
• real-business or causal conclusions from the Simulation Harness;
• distributed admission enforcement from the in-memory serving_limits state;
• safe source-query plans and acceptable impact of long consistent-read transactions.

This distinction is intentional: the code establishes semantics and safety boundaries, while the qualification harness and production telemetry establish capacity, latency, and availability.

21. Additive capabilities

The original six-strategy baseline has expanded to eleven registered trained strategies, with additional optional serving and operational capabilities. Ordinary source-free serving and atomic publication remain the shared boundaries:

• Category/geographic strategies and Swimlanes add five trained category strategies, sparse local variations, two typed recent-history candidate sources, and configured ordered composition. The contextual endpoint reads all requested variations from one head. See geography and Swimlane rationale and the current design reconciliation.

• Polars aggregation selects an alternative complete derived work store. It shares source and reduction semantics with DuckDB but has different thread/memory/scratch behavior; see section 9.

• Shopper personalization is default-off and scope/version allowlisted. Ordered interactions build a bounded, rebuildable projection; serving normally reranks only candidates from the current snapshot. Read shopper-personalization.md for the full contract. The pipeline and worker now publish snapshot-scoped category/brand/eligibility features from the consistent Catalog read before activating the head. Missing older companions safely omit those feature terms. Category/brand concentration controls can use this metadata; the current Shopper projection does not yet derive category/brand affinities from Item interactions.

• Embedded ANN retrieval is separately default-off and scope allowlisted. The default metadata_svd model projects the current metadata matrix with deterministic truncated SVD. An explicit ann.model = "two_tower" instead trains separate PyTorch query and Item encoders from bounded aggregate co-purchase/co-view pairs in the same Training Run. It uses MPS when available, otherwise CPU; no raw interaction group identifiers enter the model artifact. Both models normalize finite float32 Item vectors and build a Faiss CPU HNSW index over eligible Items. An optional recommendation_ann_artifact row holds the Product ID mapping, vectors, native index, representation version and SHA3-512 checksum under the Snapshot ID. The torch-two-tower-v1 payload additionally holds metadata input vectors and query-tower weights. API serving evaluates that small query encoder in NumPy from the published artifact; it never imports PyTorch or reads the merchant source. Metadata artifacts use metadata-svd-v3; older payloads fail closed. Publication validates the payload before creating the building snapshot, then inserts it in the same transaction that advances the serving head. The API loads only available, same-scope artifacts through a two-generation bounded cache and schedules at most one cold load off the request path; a warming, absent or invalid artifact retains the existing Similar Items and For You behavior. The profile-derived For You query can discover eligible Items beyond the old global union. Small eligible indexes and ANN underfill use exact vector search. Faiss imports only when an opted-in build or load occurs, avoiding native-library initialization in default-off workers and APIs. Neither the aggregate-pair training objective nor the representation/index is qualified for broad production enablement; aggregate co-purchases can encode complements rather than substitutes. See ann-index-experiment.md and the opt-in launch gates in PL-244.
• Commerce Simulation Harness owns a separate mutable source, storefront, bounded Synthetic Visitor Runner, checkpoint protocol, and immutable evidence report. It exercises production-shaped HTTP and adapter seams but makes no production-quality or causal claim. Read commerce-simulation-harness.md.
• Observability and quality drift expose bounded operational and aggregate model/data evidence. Drift is diagnostic and fail-open after publication; it never changes the serving head. Read recommendations-operations.md.
• Serving admission control is an optional reusable package with token-bucket and concurrency policies. The recommendation API only installs its ASGI middleware when both a controller and a trusted request-context factory are injected. The bundled in-memory state is process-local and must not be described as distributed enforcement. Read ../design/serving-admission-control.md.

For precise resource, cache, completeness, and concurrent-read limits, see the current limitation register.

These boundaries are intentionally separate: snapshots remain the authoritative aggregate serving artifact; Shopper state is additive and disposable; simulation data is local/test-only; telemetry cannot affect service correctness; and reusable admission code cannot import the application.