Commerce Recommendations repository guide¶

This is a user and developer book for the repository as it exists now. It explains the runtime, the code paths, the data model, the algorithms, the operational boundaries, and the reasons behind the important choices. It is explanatory rather than normative: when this guide disagrees with source code, migrations, executable tests, or an approved specification, those sources win. The repository's authority order is summarized in AGENTS.md and the documentation map in technical-implementation.md.

Two labels appear throughout:

Documented rationale means the reason is explicitly recorded in an approved specification, design, product decision, or operational contract.
Implementation inference means the reason follows strongly from code structure or tests but is not recorded as an approved decision. Treat it as an explanation to validate, not governance.

For the current engineering tradeoffs, newer retrieval paths, and known implementation limits, read engineering rationale. For concrete change recipes and verification choices, read implementation workflow.

1. What the product is¶

The service turns a Commerce Property's relational Catalog and Interaction Dataset into immutable, versioned Recommendation Snapshots. A separate Serving API retrieves pre-ranked Recommendation Sets from the latest successfully published Snapshot. External systems still own catalog changes, interaction capture, scheduling, ordinary caller authentication, and presentation. The product boundary is stated in the approved specification and implemented as separate API and worker processes (architecture summary).

The eleven trained Recommendation Strategies are:

Shape	Strategy	Primary evidence	Purpose
Item-to-Item	Frequently Bought Together	same-Order co-purchase	complementary Items
Item-to-Item	Also Viewed	same-Browsing-Session co-view	adjacent browsing interest
Item-to-Item	Similar Items	Item Metadata similarity	substitutes and Strict Cold Items
Global	Best Sellers	decayed purchase popularity	strong recent demand
Global	Most Viewed	decayed view popularity	broad recent attention
Global	Trending	recent momentum over a longer baseline	accelerating attention
Item-to-Item	Popular Viewed Same Category	category-local decayed views	popular peers
Item-to-Item	Popular Bought Same Category	category-local decayed purchase quantity	popular purchased peers
Item-to-Item	Frequent Bought Different Category	cross-category co-purchase	supported cross-category relationships
Item-to-Item	Frequent Viewed Same Category	same-category co-view	related browsing within a category
Item-to-Item	Frequent Viewed Different Category	cross-category co-view	related browsing across categories

Their shapes and ordered evidence tiers are one registry, not duplicated route logic (STRATEGY_DEFINITIONS). For You is different: it is a serving-time personalized strategy: normally bounded reranking, or opt-in ANN discovery from that Snapshot's eligible Catalog. It is not a trained Snapshot strategy (serving strategy enum).

The most important identity is the Commerce Scope:

(data_source_id, tracking_id, catalog_id)

data_source_id selects a configured Relational Data Source, tracking_id identifies the Commerce Property inside that source, and catalog_id selects its Catalog. Product ID is meaningful only inside that boundary. The canonical vocabulary, including terms that should not be substituted, is in CONTEXT.md. Code normalizes every external scope identifier to Unicode NFC, enforces the 255-byte storage bound, and carries all three fields in CommerceScope (domain validation, CommerceScope).

Why the boundary is this strict¶

Documented rationale. Property-local evidence prevents one independently operated Commerce Property's behavior from influencing another. Catalog locality prevents accidental cross-Catalog discovery when source Product IDs are not globally unique. These were explicit product decisions D-007 and D-011 through D-014 (product decision log) and are observable requirements, not merely implementation preferences (isolation acceptance example).

2. The system in one picture¶

flowchart LR
    Scheduler[External scheduler] -->|POST Training Run| API[API process]
    API -->|pending run| Control[(Control database)]
    Worker[Worker process] -->|claim + lease| Control
    Worker -->|consistent, scoped streams| Source[(Relational Data Source)]
    Worker -->|derived aggregates only| Duck[(Ephemeral DuckDB or Polars work store)]
    Duck --> Pipeline[Eleven-strategy pipeline]
    Pipeline -->|building Snapshot| Control
    Control -->|atomic head swap| Head[Serving head]
    Client[Downstream client] -->|GET Recommendation Set| API
    API -->|head + pre-ranked payload| Control
    API -. never reads .-> Source

The end-to-end invariant is simple: source reads belong to one Commerce Scope and one consistent cutoff; raw interaction rows are reduced while streaming; a complete Snapshot becomes visible in one atomic publication; serving reads that published state only. The concise flow is maintained in architecture.md, while the implementation is split between source.py, pipeline/run.py, storage.py, and worker.py.

Why API and worker are separate¶

Documented rationale. Training is CPU-, I/O-, and scratch-intensive, while serving has a tight latency objective. A durable worker survives caller disconnects and API restarts and prevents batch work from competing in the API process. This is design decision D-01/D-02 (design decisions). The code enforces the split: the API records a pending run (submission handler); the worker claims and executes it (TrainingWorker.run_once).

Dependency direction¶

The application follows an inward dependency shape:

domain values and pure policies
        ^
contracts / pipeline / personalization / simulation / persistence
        ^
API and worker composition roots

The reusable serving_limits package may not import recommendations. Only recommendations.config and recommendations.observability may import the sibling telemetry package. Production code may not import tests or prototypes. These are executable architecture rules in scripts/validate_architecture.py, run by make architecture-check.

Documented rationale. The admission controller is meant to be reusable by many model services, so application concepts must not leak into it. Recommendation telemetry meaning belongs to this application, while generic exporters and transport safety belong to the platform package (architecture dependency contract).

3. Repository tour¶

Path	Responsibility
`src/recommendations/domain.py`	Core identities, strategy registry, Recommendation Set and Training Run invariants
`src/recommendations/contracts.py`	Strict Pydantic HTTP request and response shapes
`src/recommendations/config.py`	Deployment JSON parsing, environment references, policy validation, configuration fingerprint
`src/recommendations/source.py`	Engine-neutral canonical streaming boundary over SQLAlchemy
`src/recommendations/pipeline/`	Reduction, algorithms, evaluation, fallbacks, and ephemeral work store
`src/recommendations/storage.py`	Durable runs, leases, snapshots, serving heads, candidate features, repository assembly
`src/recommendations/api.py`	FastAPI composition root and transport-to-domain error mapping
`src/recommendations/worker.py`	Claim/heartbeat/execute/publish/fail/retain loop
`src/recommendations/personalization/`	Authorization, interactions, profiles, lifecycle, causal tokens, reranking
`src/recommendations/synthetic/`	Deterministic Generated Relational Data Sources and manifests
`src/recommendations/simulation/`	Storefront, Synthetic Visitor Runner, checkpoints, evidence, safety controls
`src/recommendations/observability/`	Typed application telemetry seam and platform adapter
`src/serving_limits/`	Framework-neutral token-bucket and concurrency admission control
`migrations/`	Authoritative Alembic evolution of the service-owned schema
`config/`	Example deployment, simulation source, scenario, and admission policies
`operations/`	Backend-neutral dashboard, alert, and runbook contract
`tests/`	Unit, contract, integration, PostgreSQL, and delivery evidence
`scripts/`	Verification, OpenAPI export, qualification, checksums, and evidence assembly

Console entry points are declared in pyproject.toml:

recommendations-api             -> recommendations.api:main
recommendations-worker          -> recommendations.worker:main
recommendations-generate-source -> recommendations.synthetic.cli:main
recommendations-simulation      -> recommendations.simulation.cli:main

Python 3.12 through 3.14 is supported, with Python 3.14 used by repository lint/type settings and the container image (package metadata, Dockerfile).

4. Running the repository¶

4.1 Developer setup¶

The supported command surface is the Makefile, not hand-assembled tool invocations:

make setup
make doctor
make test-focused TEST=tests/unit/test_cooccurrence.py::test_name
make test
make smoke

make setup asks uv to install the locked development, PostgreSQL, observability, and PyTorch extras. make doctor checks the expected toolchain and sibling telemetry checkout. The full ladder is in docs/testing.md and encoded in the Makefile.

The sibling repository layout matters because development resolves telemetry as an editable path:

workspace/
├── recommendations/
└── telemetry/

That source override is declared in pyproject.toml. Published wheels still declare a normal compatible telemetry version, so the sibling checkout is a development and CI resolution mechanism, not a runtime filesystem contract.

4.2 Run API and worker directly¶

Create a deployment file from config/data_sources.example.json, then set the control database, configuration file, and each source URL through environment variables:

export RECOMMENDATIONS_CONTROL_DATABASE_URL='postgresql+psycopg://...'
export RECOMMENDATIONS_DATA_SOURCES_FILE="$PWD/config/data_sources.json"
export MERCHANT_PLATFORM_DATABASE_URL='postgresql+psycopg://...'

.venv/bin/alembic upgrade head
.venv/bin/recommendations-api

In a second process:

.venv/bin/recommendations-worker

The two required bootstrap variables and strict JSON loading are enforced in load_deployment_config. Source credentials remain in the environment named by connection_url_env; the JSON stores only the reference (DataSourceConfig.connection_url).

4.3 Run the complete local simulation¶

docker compose up --build
docker compose --profile runner run --rm simulation-runner

This starts two PostgreSQL databases, migrations, simulation initialization, API, worker, storefront, and optionally the one-shot runner. The control database and mutable Simulation Data Source are deliberately separate (Compose services). The API is exposed on port 8000 and the storefront on 8080 by default.

Documented rationale. The simulation must exercise the normal Training API, worker, Data Source Adapter, and Serving API. It is not allowed a privileged ingestion or Snapshot mutation path (harness boundary). Separate databases make that boundary visible and keep test-commerce mutation out of the service-owned store.

5. Configuration as a reproducibility contract¶

Deployment configuration has these top-level groups:

data_sources: source IDs, allowed scopes, secret environment references, isolation level, and ordered canonical SQL;
pipeline: aggregation/counting backend selection, resource bounds, group limits, and parameter grids;
ann: scope allowlist, representation model, and dimension budget;
swimlanes: scope-bound names, quotas, and ordered strategy/geography/filter steps;
freshness: default and per-scope stale windows;
retention: Snapshot, Training Run, and failure diagnostic lifetimes;
quality_drift: comparison policy and thresholds;
personalization: default-off capabilities, principals, roles, policies, and secret reference;
observability: process logging, OTLP transport, instrumentation, and signal bounds;
worker_poll_seconds: worker idle cadence.

Unknown fields fail closed at every parsed level; for example, the root allowlist is explicit in load_deployment_config. This catches misspelled or unsupported operational intent rather than silently ignoring it.

The service computes config_version as a canonical SHA3-512 digest of the behavior-affecting, non-secret configuration (fingerprint builder). A run also records the implementation model_version and Training Evidence Schema version (schema identities).

Documented rationale. A run must be reproducible and comparable across workers. Persisting a content identity rather than a friendly label prevents two different query/policy definitions from sharing one version accidentally. Secrets are excluded so the fingerprint is stable across secret rotation and does not become credential material.

The source queries are deployment-owned, reviewed SQL. Runtime values are bound parameters. The caller cannot submit SQL. This supports different physical schemas and SQLAlchemy dialects while keeping the application-facing rows stable (example mappings, canonical query contract).

6. Public API contracts¶

6.1 Submit and inspect a Training Run¶

curl -X POST http://127.0.0.1:8000/v1/training-runs \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: daily-2026-09-21' \
  -d '{
    "data_source_id": "merchant-platform",
    "tracking_id": "store-42",
    "catalog_id": "main"
  }'

The API returns 202 with a durable run_id and pending status. It validates the complete scope against the registry before creating state. An exact same-scope idempotency replay returns the existing run; a different key while that scope already has a pending or running run returns 409 training_run_active with its ID. The behavior lives in the submission handler and database uniqueness constraints (storage indexes).

curl http://127.0.0.1:8000/v1/training-runs/RUN_ID

The status response includes timestamps, cutoff, Snapshot ID, safe failure category, aggregate metrics, configuration/model versions, and ordered lifecycle transitions (response model). Training states are deliberately small: pending -> running -> succeeded|failed (status enum).

6.2 Serve Recommendation Sets¶

Anchored strategies use an Item path:

curl 'http://127.0.0.1:8000/v1/catalogs/main/items/SKU-1/recommendations/also-viewed?data_source_id=merchant-platform&tracking_id=store-42&limit=20'

Global strategies omit the Item:

curl 'http://127.0.0.1:8000/v1/catalogs/main/recommendations/trending?data_source_id=merchant-platform&tracking_id=store-42&limit=20'

The response describes the exact Snapshot and cutoff, stale state, optional insufficient_evidence reason, and ranked Items with score, confidence, support, and Recommendation Provenance (response contracts). limit defaults to 20 and is bounded from 1 through 100 at the HTTP layer (anchored route, global route).

Representative response:

{
  "strategy": "also-viewed",
  "anchor_product_id": "SKU-1",
  "reason": null,
  "snapshot_id": "…",
  "generated_at": "2026-09-21T12:05:00Z",
  "data_cutoff": "2026-09-21T12:00:00Z",
  "is_stale": false,
  "items": [
    {
      "product_id": "SKU-2",
      "rank": 1,
      "score": 0.73,
      "confidence_tier": "high",
      "provenance": {
        "source_tier": "co_view",
        "evidence_type": "co_view",
        "support": 18,
        "fallback_reason": null
      }
    }
  ]
}

This example illustrates the contract; its values are not fixture expectations.

Serving error semantics distinguish no published Snapshot (404 snapshot_not_found), an anchor not present in the head Snapshot (404 item_not_found), invalid shape/scope (422), and detected Snapshot corruption (503 snapshot_incomplete). An empty but valid Recommendation Set is still 200, with reason: "insufficient_evidence". Staleness is metadata, not an outage: an older head continues serving with is_stale: true. These cases are executable in tests/contract/test_api.py and specified by acceptance examples EX-010 through EX-016 (specification).

Why ordinary serving uses precomputed lists¶

Documented rationale. A bounded pre-ranked payload means ordinary serving performs one indexed head/set lookup and a slice, supporting predictable latency and keeping merchant source failures out of the request path. Storing one bounded JSON payload per (snapshot, strategy, anchor) also avoids the row/index multiplication of one row per candidate (D-13 in the design). The repository lookup joins through serving_head and reconstructs domain values without calling a source adapter (get_head_set).

Optional personalization, recent-seed fusion, and ANN retrieval do compute at request time. They remain bounded and read published artifacts/profiles rather than the merchant source. See the current serving path comparison.

7. The Relational Data Source boundary¶

The adapter exposes a logical schema instead of imposing a physical one:

class SourceAdapter(Protocol):
    def open(
        self,
        scope: CommerceScope,
        history_start: datetime,
        cutoff: datetime,
    ) -> SourceRead: ...

class SourceRead(Protocol):
    consistency_token: str
    def stream_catalog(self, fetch_size: int = 10_000) -> Iterator[CatalogRow]: ...
    def stream_views(self, fetch_size: int = 10_000) -> Iterator[ViewRow]: ...
    def stream_purchases(self, fetch_size: int = 10_000) -> Iterator[PurchaseRow]: ...

This is an abbreviated extract of the actual protocols (source interfaces).

The canonical row model requires:

Stream	Required semantics	Required order
Catalog	scope, Product ID, Recommendation Eligibility; optional category/brand/price/text/time	Product ID
views	scope, Product ID, Browsing Session ID, event time	session, time, Product ID, optional source sequence
online purchases	scope, Product ID, Order ID, time, quantity	Order, time, Product ID, optional sequence
offline purchases	same as online; entire stream optional	same, after online stream
compatibility	anchor and candidate in scope; optional stream	anchor, candidate

Canonical row dataclasses are in source.py. PurchaseRow.group_id prefixes the Purchase Channel so an online and offline Order with the same source Order ID cannot create false co-purchase evidence (channel-scoped grouping).

SqlAlchemySourceAdapter opens one repeatable-read transaction (serializable for SQLite), binds tracking_id, catalog_id, history_start, and exclusive cutoff, streams partitions, validates every row's scope/window/order, and rolls the read transaction back on exit (session construction, stream validation).

Why a consistency token exists¶

Documented rationale. A shared cutoff does not prevent a late/backdated write from changing a later query in the same run. One transaction or engine time-travel point must therefore cover Catalog, views, purchases, and compatibility. The opaque token makes accidental mixing of read sessions detectable (D-05 in the design); the pipeline asserts it after each stream (orchestration).

Why raw rows are never staged¶

Documented rationale. Direct streaming is an approved privacy and ownership boundary: the service is allowed to derive Recommendation evidence, not become another raw commerce-data lake. It also limits breach and retention scope. The contract suite forbids a raw staging path (test_no_raw_staging.py), and the DerivedWorkStore schema contains only pair/support/popularity/category deltas and exclusion counts (derived schema).

The Catalog itself is materialized in memory because feature construction and the full set of eligible anchors need it; the large Interaction Dataset is streamed once and reduced. This distinction is explicit in _load_catalog and DerivedWorkStore.consume_*.

8. Training, step by step¶

8.1 Durable claim and cutoff¶

The API inserts a pending Training Run.
A worker selects the oldest claimable run and writes running, started_at, a fixed data_cutoff, lease owner, and lease expiry.
A heartbeat renews ownership while training runs.
The worker reads the preceding two years from the configured adapter.

The main loop is visible in TrainingWorker.run_once. The database claim supports PostgreSQL row locking/skip-locked behavior and bounded recovery of an expired lease (TrainingRunRepository.claim_next).

Documented rationale. The database-backed queue avoids introducing another service while still making idempotency, one-active-run-per-scope, and recovery transactional (D-11 in the design).

8.2 Stream into bounded derived state¶

generate_recommendations fixes a 28-day chronological holdout, loads Catalog eligibility, reduces views and purchases into the configured ephemeral aggregate work store, constructs providers, builds full-data publication sets, builds holdout-safe evaluation sets, and returns aggregate metrics (pipeline entry point).

The work store configures memory, threads, and a maximum spill directory, flushes aggregate maps when their distinct-key limit is reached, and deletes the temporary directory on every context exit (work-store lifecycle, bounded flush).

Documented rationale. DuckDB supplies out-of-core aggregation without persisting canonical raw rows. Per-run scratch prevents evidence mixing and makes cleanup deterministic. This is D-06; the single-node worker remains replaceable if qualification later proves it insufficient. The optional Polars engine uses derived-only Parquet and its own execution controls; DuckDB memory/spill settings must not be assumed to provide identical guarantees there. Backend parity and resource distinctions are explained in the engineering guide.

8.3 Co-occurrence semantics¶

Within a Browsing Session or Order, duplicate Product occurrences contribute once to pair/support evidence. Views still add one popularity unit per row; purchases add quantity. For Items a and b, behavioral score is:

cosine = pair_support / sqrt(support(a) * support(b))
score  = cosine * pair_support / (pair_support + shrinkage)

Pairs below min_support are absent. The implementation is regularized_cosine_score.

Sessions and Orders above their configured distinct-Item limits are excluded whole from pair, support, and category-pair evidence. The defaults are 100 per session and 200 per Order. Their rows still contribute popularity and Trending. The reducer deliberately decides only after seeing the complete group (limits and reducer, group completion).

Documented rationale. Pair generation is quadratic in distinct group size. Whole-group exclusion avoids a partial, order-dependent truncation while retaining useful popularity evidence; the excluded group/row counts remain observable. This is product decision D-017 (decision log).

8.4 Similar Items¶

Eligible Item Metadata is encoded as sparse features: category, brand, price bucket, and word/character TF-IDF from title/description. Rows are L2-normalized and sparse-dot-topn computes a bounded cosine neighborhood rather than a dense Catalog-by-Catalog matrix (content implementation).

Documented rationale. Structured/text metadata is deterministic, reproducible, and deployable without a network model service. The design rejected embeddings for V1 because their model distribution and inference complexity were not justified by local evidence (D-09 and alternatives).

Best Sellers and Most Viewed use exponential half-life decay:

weighted contribution = weight * exp(-ln(2) * age_days / half_life_days)

Trending compares a recent window to a longer baseline with smoothing, then combines view and purchase momentum with configured weights (popularity functions, decay implementation). Parameter grids in the deployment file let evaluation select view/purchase half-lives, recent/baseline windows, and alpha without silently searching an unrecorded space.

8.6 Evaluation and parameter selection¶

Whole shopping groups are placed into train or holdout based on their maximum event time, so one Order or Browsing Session cannot leak across the temporal boundary (partition logic). Ranking quality uses Recall@K, NDCG@K, catalog coverage, anchor count, and support cohorts (evaluation model).

Behavioral parameter selection maximizes NDCG, then recall, then coverage, with stable conservative tie breakers. If the holdout has no usable anchors it returns recorded defaults (behavioral selection). Popularity and Trending use equivalent holdout searches over explicit grids (popularity selection). The published candidates are rebuilt from all evidence through the fixed cutoff after selection.

Documented rationale. Chronological holdout prevents future-to-past leakage; an explicit grid makes tuning evidence reproducible; rebuilding uses all legitimate source history for the final Snapshot (D-10 in the design).

8.7 Fallbacks, provenance, and stable ranks¶

Evidence types are never blended into one incomparable numeric scale. Providers are traversed in the strategy's declared tier order. Inside a tier, candidates sort by descending score and then ascending Product ID. The ranker removes the anchor, ineligible Items, and duplicates, and records the selected evidence tier and fallback reason (fallback ranker).

The effective chains are:

Frequently Bought Together: co-purchase -> category-pair purchase -> compatibility rule -> category recent purchases -> property recent purchases.
Also Viewed: co-view -> metadata similarity -> category recent views -> property recent views.
Similar Items: metadata similarity only.
Best Sellers, Most Viewed, Trending: their one global evidence type.

The registered chains are source-controlled in domain.py; provider assembly is in pipeline/run.py.

Documented rationale. Tier-first filling prevents a score from one evidence family being treated as numerically comparable with another. Recommendation Provenance makes the primary versus fallback reason observable to consumers and evaluation (D-08 in the design).

Every eligible anchor receives each anchored strategy, even when empty. Empty is represented as insufficient_evidence; it is not a missing row or a failed strategy. Recommendation Set invariants cap entries at 100, require contiguous one-based ranks, forbid duplicates/self-recommendation, and require insufficient-evidence sets to be empty (domain invariants).

9. Snapshot publication and serving consistency¶

The durable publication sequence is:

Validate that the completed manifest contains every registered trained strategy (currently eleven).
Insert a Snapshot in building state.
Insert bounded Recommendation Set payloads and candidate feature companions in chunks while the Snapshot is invisible.
Validate required sets and the worker's live lease.
In one transaction, insert the optional validated ANN artifact, mark the Snapshot available, replace the scope's serving_head, and mark the Training Run succeeded.
On any exception, delete the unpublished building Snapshot.

This is implemented in SnapshotRepository.publish.

stateDiagram-v2
    [*] --> Building
    Building --> Available: complete manifest + valid lease + atomic head swap
    Building --> Discarded: any load or validation failure
    Available --> ServingHead: pointer references Snapshot
    ServingHead --> Available: newer publication replaces pointer only

Documented rationale. Chunking avoids one enormous transaction, but a building state keeps partial chunks invisible. The final activation transaction provides all-or-nothing publication; an optional ANN payload is inserted there too and increases its size. A failed new run therefore cannot replace the last successful head (D-12 in the design).

Staleness is calculated from generated_at + freshness_hours; it never triggers synchronous training. Retention runs after successful publication, keeps the newest configured count, and always preserves the current head (snapshot retention). Defaults are five Snapshots per scope, 90 days of terminal run metadata, and 30 days of failed diagnostics (retention config).

10. Durable storage and migrations¶

PostgreSQL is the production control store; SQLite exists for focused and local tests. SQLAlchemy defines runtime tables, and Alembic defines deployable evolution. JSON uses PostgreSQL JSONB when available and generic JSON elsewhere (storage metadata).

Core tables:

Table	Meaning	Important key/constraint
`training_run`	lifecycle, cutoff, versions, lease, metrics, terminal outcome	run ID; unique scope + idempotency key; partial unique active scope
`run_failure_diagnostic`	bounded classified failure occurrence	belongs to one run
`recommendation_snapshot`	immutable build/publication identity and manifest	one Snapshot per run
`recommendation_set`	one bounded pre-ranked payload	Snapshot + strategy + anchor key + geographic level/key
`serving_head`	current available Snapshot pointer	one row per Commerce Scope
`recommendation_candidate_feature`	Snapshot-scoped non-sensitive reranking metadata	Snapshot + Product ID
`recommendation_ann_artifact`	representation, eligible mapping, vectors/index, checksum, optional query model	Snapshot ID

Personalization adds:

Table	Meaning
`shopper_interaction`	retention-bounded ordered ledger
`shopper_interaction_idempotency`	exact replay/conflict record
`shopper_profile`	compact rebuildable projection
`shopper_suppression`	opt-out/deletion barrier
`shopper_deletion_receipt`	durable lifecycle evidence

The complete runtime schema is declared in storage.py. The migration history begins with run and Snapshot state, adds evidence schema versions/runtime identities, and then adds personalization state (initial migration, personalization migration). Subsequent migrations add typed recent projections, geographic set identity, and snapshot ANN artifacts.

Documented rationale. PostgreSQL transactions, partial unique indexes, row locks, and JSONB make coordination and head publication explicit. SQLite does not substantiate the production concurrency or availability claim (D-03 in the design). Run migrations against an explicitly disposable database with make migration-check; PostgreSQL concurrency semantics require TEST_CONTROL_DATABASE_URL and make test-postgres.

11. Shopper personalization¶

Personalization is additive, scope-gated, and off by default. It does not retrain or mutate a Recommendation Snapshot. It reads a bounded Shopper Profile and normally reorders eligible candidates from the current Snapshot. Scope-enabled ANN can additionally retrieve eligible Items beyond the global candidate union from that same generation. The full operational contract is shopper-personalization.md.

11.1 Trust and interaction flow¶

sequenceDiagram
    participant C as Commerce Property
    participant A as API
    participant P as Authorization policy
    participant S as Shopper state repository
    C->>A: bearer + signed Personalization-Context + ordered interaction
    A->>P: authenticate principal, scope, role, policy
    P-->>A: bounded authorization decision
    A->>S: append ledger + project profile in one transaction
    S-->>A: sequence + profile version
    A-->>C: acknowledgement + short-lived causal token

The authenticated Commerce Property is derived from the bearer principal, not a body field. A signed context binds Data Source, Commerce Property, Catalog, opaque Shopper ID, authorization, and expiry. Authorization is checked before state access (authorization codec and policy, API pre-body authentication).

POST /v1/personalization/interactions accepts a closed event vocabulary, exact next sequence, and bounded Item list (request model). The repository commits idempotency, ordered ledger, lifecycle barrier, and profile projection transactionally. An exact retry returns the original commit; a reused ID with different content or a sequence gap is a typed conflict (transaction boundary). Only after the durable commit does the service issue an acknowledgement and causal token (interaction service).

Documented rationale. Ordered interactions make projection state deterministic; idempotency makes retries safe; acknowledging only after commit prevents clients from receiving a capability for state that may not exist. The causal token gives explicit read-your-actions behavior without exposing a Shopper ID or profile version as a bearer credential.

11.2 Serving-time reranking¶

Existing lanes opt in with personalize=true. for-you implies personalization and uses a deduplicated union of Best Sellers, Most Viewed, and Trending from the same head Snapshot; ordinary fallback is Best Sellers (global candidate lookup, personalized serving branch). At most 100 candidates are read, and the API also attempts to load their Snapshot-scoped category/brand/eligibility companion features.

The pipeline now returns Snapshot-scoped category/brand/eligibility features from its consistent Catalog read, and the worker passes them to SnapshotRepository.publish. Features are staged before activation and cleaned up on failed publication or retention. This is verified by candidate feature publication tests. Older snapshots and missing metadata retain safe defaults. The current Shopper projection does not derive category/brand affinity maps from Item interactions, so publishing candidate metadata enables concentration controls without automatically populating those affinity signals.

For You may use the separately enabled ANN path to retrieve eligible Items beyond the global union. The fallback remains the ordinary Snapshot path when an artifact is absent, invalid, warming, or cannot supply a usable query. See ANN rationale.

The deterministic reranker combines normalized Snapshot rank with Item, category, and brand affinity, then subtracts seen, negative, and recent-Item penalties. It applies category, brand, and recent-Item concentration caps. Crucially, it replaces rank only: Product IDs, original scores, confidence, and Recommendation Provenance survive (configuration and scoring).

If authorization is absent, history is insufficient, a Shopper opted out or was deleted, or state is unavailable, the API returns an ordinary Recommendation Set with a bounded personalization outcome. A valid causal token must observe at least its profile version within the configured 100 ms wait or the API returns 409 causal_not_observed; it never silently returns an older personalized order (application service). The public personalization member exposes outcome/fallback/causal observation only (safe response).

11.3 Privacy lifecycle¶

Profiles are bounded projections, not source-of-truth customer records. Recent Items are capped at 100 and affinity/count maps are capped at 256 persisted keys (state bounds, projection bounding). Opt-out and deletion create barriers so older/later writes cannot recreate suppressed state accidentally. Deletion completion and expiry run in bounded worker maintenance independently of Training Runs (worker maintenance).

Implementation inference. Running lifecycle maintenance before each worker poll reuses the durable worker without coupling cleanup to successful training. It also means cleanup cadence is linked to worker availability and polling; production sizing should validate that this cadence meets the intended privacy deadline.

12. Quality and drift¶

Each successful run retains bounded numeric evidence. Optional quality drift compares a run only to recent Comparable Training Runs for the same scope whose model, configuration, and evidence schema versions match. The assessment reports within_expected_range, warning, or critical when comparable evidence is usable, and explicitly reports insufficient_history, insufficient_sample, incomparable_version, or unavailable otherwise. It uses absolute, relative, and robust-deviation thresholds by measure family (quality statuses and entry point).

The worker performs drift assessment after publication and reports it through the observability adapter (worker phase, _assess_quality). Publication and serving are not rolled back by a drift result.

Documented rationale. Drift evidence diagnoses changes in training inputs and Recommendation outputs. It does not establish shopper outcomes, causal uplift, revenue, or conversion. Keeping it non-blocking avoids equating an unusual distribution with an invalid Snapshot; the operator runbook requires checking comparable versions and sample sufficiency (quality drift runbook).

13. Deterministic generated sources¶

recommendations.synthetic creates a reproducible Generated Relational Data Source without importing production data. A GenerationConfig fixes the Commerce Scope, profile, seed, cutoff, and source capabilities. Named RNG substreams isolate generation concerns so changing one concern need not perturb every table (generator configuration).

Profiles are fixed contracts:

Profile	Catalog	Views	Purchases	History	Categories
`smoke`	1,000	25,000	10,000	90 days	20
`development`	25,000	50,000,000	2,000,000	365 days	250
`qualification`	200,000	100,000,000	5,000,000	730 days	1,000

The definitions live in profiles.py.

recommendations-generate-source \
  --profile smoke \
  --output-dir artifacts/generated-smoke

Generation produces:

a Generation Manifest with resolved profile, lineage, counts, planted signals, per-table logical SHA3-512 digests, and content-derived dataset ID;
an Oracle Manifest with bounded expected observations;
after SQLite or PostgreSQL materialization, a receipt that re-reads counts/digests and verifies the realization.

PostgreSQL materialization reads its connection URL indirectly from a named environment variable:

recommendations-generate-source \
  --profile development \
  --database-url-env GENERATED_SOURCE_DATABASE_URL \
  --workers 4 \
  --output-dir artifacts/generated-development

The PostgreSQL path uses bounded concurrent psycopg COPY workers, parallel per-table index and digest work, and uniquely named staging tables. --workers defaults to 4 (maximum 16); a final transaction publishes only after every receipt check passes, preserving the prior source on failure. The CLI generates each table once and hashes its rows as it feeds COPY; the database readback independently verifies the result. Tables load in Catalog-first order, so the largest table's serial producer and verifier remain performance limits. Direct callers can use provision_postgresql(config, destination) for this one-pass path or materialize_postgresql(source, destination) for an already manifested source. The generated source guide covers CLI controls, Docker setup, performance evidence, and failure handling.

Manifest schemas and canonical hashing are in manifest.py; engine materializers are in sqlite.py and postgresql.py. Raw generated interaction rows do not enter the evidence documents.

Documented rationale. A logical manifest separates reproducibility from a database engine. A materialization receipt proves that one realization matches that manifest; it does not alone qualify PostgreSQL capacity or concurrency. The normal Source Adapter remains the only training ingestion seam (generated source guide).

14. Commerce Simulation Harness¶

The harness adds mutable shopping activity around the otherwise batch-oriented system:

Generated seed -> Simulation Data Source -> storefront/visitors -> checkpoint
              -> ordinary Training API -> worker -> ordinary Serving API -> evidence report

One stack supports one fixed Commerce Scope and one non-terminal Simulation Run. The Simulation Data Source owns catalog rows, views, Online Purchases, compatibility rules, session metadata, run lineage, and idempotent write records (source model, database service). Once the source is mutated by simulated shopping it is no longer a verified Generated Relational Data Source.

14.1 Scenarios and visitors¶

Scenario JSON is strict and versioned. Resolution combines authored values, versioned defaults, an allowlist of web overrides, and non-overridable deployment safety caps. Logical IDs, timestamps, mission choice, action plans, and checkpoint layout derive from the scenario digest and root seed (scenario models and resolver).

Independent-Behavior Scenarios do not let served recommendations influence future visitor actions. Recommendation-Feedback Scenarios may follow served results and must label the feedback loop. Their recommendation-dependent ranking metrics are unavailable rather than presented as independent quality evidence (scenario modes, feedback safeguards).

Documented rationale. Determinism requires logical inputs to be reproducible, but operational facts such as process UUIDs, scheduler order, wall time, and request latency are still recorded rather than forced. Distinguishing independent behavior from feedback prevents the service from grading itself on traffic it caused.

14.2 Checkpoints¶

A checkpoint closes admission, drains in-flight writes, captures source counts, submits one deterministic Training Run idempotency key, polls that exact run, and verifies its Snapshot through the Serving API. It never reads the control database, selects a cutoff, cancels a worker, or silently substitutes another run (orchestrator).

Documented rationale. The checkpoint must measure the public service contract, including its asynchrony and failure behavior. Reaching into storage would create a test-only success path and could hide API/worker integration defects.

14.3 Evidence and safety¶

The authoritative report is versioned JSON; HTML only projects stored values. It includes identity, reconciled visitors/actions, source deltas, bounded latency/error histograms, Training Run/Snapshot attribution, coverage/fallback/provenance/concentration, planted-mission rankings where valid, feedback distributions, explicit unavailable reasons, and capped diagnostics (evidence schema).

The harness starts only in local or test, validates hosts/schemes/ports/resolved addresses and redirects, rejects credential-bearing URLs, and caps traffic, retries, response size, diagnostics, feedback, and artifacts (target policy). Active-run reset is rejected; cancellation closes admission and drains boundedly.

The evidence proves only behavior under its synthetic scenario. It does not prove production capacity, conversion, revenue, causal uplift, or real-Shopper relevance (harness guide).

15. Observability¶

Application code emits typed operations, metrics, and events through a narrow Observability port. Metric attributes are closed enums: route template, HTTP method/status class, operation, strategy, phase, outcome, failure category, and source kind. Scope, Product, run, and Snapshot IDs may be correlation context for traces/events but are excluded from metric dimensions (observability vocabulary).

FailSafeObservability prevents an exporter defect from changing application behavior. When observability is disabled, the runtime provides a no-op implementation. When enabled, the adapter builds the sibling telemetry runtime, instruments FastAPI/SQLAlchemy, and owns flush/shutdown (runtime wrapper).

Documented rationale. The service owns the meaning and privacy classification of its signals; the shared telemetry package owns generic SDK/runtime/export mechanics. Metrics need bounded cardinality, while sensitive identifiers belong only in access-controlled correlation channels. Telemetry is diagnostic and therefore fail-open with respect to the serving data plane (operations contract).

The backend-neutral dashboards, alerts, and response procedures are source-controlled in operations/recommendations.toml. Operator investigations should start with build/config/collector identity, then separate request, database, source, publication, and telemetry failure domains.

16. Reusable serving admission control¶

serving_limits is an optional package. A policy resolves all matching hierarchical token-bucket and concurrency rules from trusted descriptors. Acquisition is all-or-nothing; an allowed decision returns an expiring lease, and release is idempotent (core types, in-memory state). The ASGI middleware is installed only when both a controller and trusted request-to-context factory are supplied (API assembly).

Documented rationale. Admission identity must come from authenticated/routing context, never raw client claims. Concurrency protects expensive in-flight work when latency rises; token buckets control rate and burst. All-or-nothing acquisition prevents one denied rule from consuming another rule's capacity. TTLs recover leaked permits after process failure (admission design).

The included InMemoryLimitState is suitable for tests or one process only. It is not a distributed fleet-capacity claim. The design still requires a production state adapter, policy publication, gateway integration, and load qualification (distributed adapter gap).

17. Failure model¶

Failure	Durable/client effect	Why it is safe
unknown source or scope	submission/serving `404`	no unregistered boundary is inferred
another active run	`409` with active run ID	database enforces one pending/running run per scope
invalid/out-of-scope/unordered source row	run fails `source_invalid`	bad evidence is not partially accepted
source consistency/incomplete stream	run fails `source_incomplete`	mixed/incomplete cutoffs are not published
DuckDB out of memory	run fails `resource_exhausted`	no partial Snapshot becomes visible
computation exception	run fails `strategy_computation_error`	complete registered-strategy manifest is mandatory
lease loss	old worker cannot publish	publication rechecks owner and expiry transactionally
failure during build/publication	building rows discarded	previous serving head remains unchanged
no successful Snapshot	serving `404 snapshot_not_found`	serving never trains implicitly
missing/corrupt required set	serving `503 snapshot_incomplete`	corruption is distinct from insufficient evidence
stale Snapshot	`200` with `is_stale: true`	availability preserved with explicit freshness
personalization unavailable/unauthorized	ordinary bounded fallback	base Snapshot remains useful and response is privacy-safe
causal watermark not observed	`409 causal_not_observed`	stale personalized state is never silently used

Worker exception mapping is explicit in run_once. Failed transitions record only a bounded category and schema version (mark_failed). The corresponding operational procedures are in recommendations-operations.md.

18. Security, privacy, and resource invariants¶

When changing code, preserve these properties:

Scope every source row, run, Snapshot, head, profile, and serving request by Data Source, Tracking ID, and Catalog ID.
Never combine behavioral evidence across Commerce Properties or Catalogs.
Serving reads only a published Snapshot and, when requested, bounded personalization projections.
Never stage, log, export, fixture, or retain raw production interaction rows.
Never log credentials, bearer material, raw Shopper IDs, raw Order IDs, raw Browsing Session IDs, request bodies, or Recommendation payloads.
Keep result lists, groups, queues, leases, retries, request/response bodies, telemetry dimensions, diagnostics, scratch, profile features, and evidence artifacts bounded.
A failed run cannot move serving_head.
A Deployment Limit Policy may tighten a Model Capability Contract but may not relax it.
serving_limits stays reusable and independent of recommendations.
Only configuration and the observability adapter may import telemetry.

These are consolidated in AGENTS.md, implemented across the source and storage boundaries, and protected by contract and architecture tests.

19. How tests establish behavior¶

Tests are organized by the boundary they prove:

Layer	What it proves	Representative files
Unit	algorithms, policy, invariants, deterministic state	`test_cooccurrence.py`, `test_fallbacks.py`, `test_personalization_core.py`
Contract	HTTP shapes/errors, source streaming, privacy and import seams	`test_api.py`, `test_source_adapter.py`, `test_no_raw_staging.py`
Integration	cross-component lifecycle, migrations, persistence	`test_end_to_end.py`, `test_lifecycle.py`, `test_personalization_storage.py`
PostgreSQL	real locking, generated-source verification, and atomic publication	`test_postgresql_storage.py`, `test_postgresql_synthetic_source.py`
Delivery	repository command, packaging, CI, agent-readiness contracts	`test_repository_harness.py`, `test_verification_workflow.py`

The normal workflow is narrow-to-wide:

make test-focused TEST=tests/path.py::test_name
make test-static
make test-unit
make test-contract
make test-integration
make test
make smoke
make verify

Use make migration-check after migration changes and make compose-check after container or Compose changes. PostgreSQL tests refuse to run without an explicitly configured disposable TEST_CONTROL_DATABASE_URL (Make targets). make smoke builds a wheel and sdist, creates a clean environment, installs the wheel without the source tree, and verifies imports and all console entry points (installed-artifact target).

Documented rationale. A passing source-tree test can hide missing package data, undeclared dependencies, or broken entry points. Clean-wheel smoke testing exercises what will actually be distributed. PostgreSQL-only tests are separated because SQLite cannot prove the locking and partial-index semantics relied on in production.

CI runs on a self-hosted Apple Silicon macOS runner, checks out the pinned telemetry repository as a sibling, tests the authoritative Python version, and checks the installed wheel across Python 3.12, 3.13, and 3.14. The exact hosted contract and limitations are in testing.md and verification.yml.

20. A practical change guide¶

Add or change a Recommendation Strategy¶

Decide whether it is anchored or global and add its evidence order to STRATEGY_DEFINITIONS.
Implement pure candidate generation under pipeline/.
Add providers in _build_candidate_providers and ensure _build_recommendation_sets emits it for every required anchor/global key.
Extend evaluation and aggregate evidence without staging raw rows.
Update Snapshot completeness, API enum/shape, observability vocabulary if needed, and tests at unit, contract, and integration layers.
Update the approved specification/design before changing consumer-visible intent.

The central registry reduces the chance that API shape, metrics prefix, fallback order, and publication completeness drift apart.

Add a Relational Data Source dialect¶

Keep the canonical row contract unchanged.
Supply reviewed ordered queries and secret-backed connection URL.
Prove one consistent read across all streams for the dialect.
Run the shared source contract suite, including scope, time window, ordering, completeness, and no-staging behavior.
Do not claim production compatibility from “SQLAlchemy can connect”; certification depends on transaction and streaming semantics.

Change persistence¶

Update SQLAlchemy runtime metadata and add an Alembic migration; never edit an applied migration to describe a new state.
Preserve downgrade behavior where the repository's migration contract requires it.
Add migration round-trip tests and run make migration-check.
If publication or leases change, run PostgreSQL tests because transaction semantics are the feature being changed.
Update public/durable schema documentation and schema version constants where meaning changes.

Change personalization¶

Start from authorization, scope, lifecycle, retention, and response-privacy invariants.
Keep candidate generation Snapshot-only and bounded; do not query the merchant source at serve time.
Preserve exact idempotency and ordered sequence conflict behavior.
Treat causal-token behavior and fallback disclosure as public API contracts.
Update offline evaluation and simulation journeys without presenting feedback traffic as independent evidence.

Change observability¶

Add meaning to the application seam and operations contract first.
Keep metrics on closed low-cardinality dimensions.
Route platform SDK/export behavior through recommendations.observability.
Verify privacy canaries and failure isolation; telemetry may not change domain outcomes.

21. Rationale ledger¶

This table is a compact ownership aid. “Recorded” means an explicit repository decision; “inferred” means this guide is reading intent from implementation structure.

Choice	Reason	Evidence status
Property/Catalog-local behavior	source IDs are scoped and cross-property evidence is prohibited	Recorded: product decisions and specification
Batch snapshots rather than online training	predictable serving, source isolation, asynchronous durable work	Recorded: design D-01/D-02/D-12
SQLAlchemy canonical adapter	support different relational engines/physical schemas behind one contract	Recorded: product D-009/D-010 and design D-04
consistent transaction/token	cutoff alone cannot prevent late/backdated multi-query inconsistency	Recorded: design D-05
derived-only DuckDB	bounded out-of-core aggregation without raw staging	Recorded: design D-06
whole oversized-group exclusion	prevent quadratic explosion and order-biased truncation	Recorded: product D-017/design D-07
regularized cosine	normalize popularity and shrink weak co-occurrence	Recorded: design D-08; formula in code
sparse metadata cosine	deterministic cold-item support without model-service dependency	Recorded: design D-09
chronological evaluation	prevent temporal leakage and record tuning space	Recorded: design D-10
database queue/leases	transactional idempotency, conflict, claim, and recovery without another service	Recorded: design D-11
invisible build + atomic head	chunk large writes while guaranteeing all-or-nothing serving	Recorded: design D-12
one JSON payload per set	one bounded lookup and far fewer rows than candidate-per-row storage	Recorded: design D-13
stale serving	prioritize availability while disclosing freshness	Recorded: specification and design D-14
default-off personalization	preserve base API behavior and require scope/version rollout	Recorded: personalization contract
default bounded rerank; opt-in snapshot ANN	preserve generation eligibility and source-free serving while allowing explicit discovery	Recorded: personalization and ANN contracts
worker-driven personalization cleanup	reuse an existing durable background loop	Inferred; validate cadence operationally
typed observability seam	keep domain signal meaning local and exporter mechanics reusable	Recorded: architecture/operations contract
independent simulation mode	avoid treating recommendation-influenced behavior as independent quality evidence	Recorded: harness contract
optional admission middleware	reuse one core while production gateway/distributed state remain unqualified	Recorded: admission-control proposal

22. What this repository does not yet prove¶

Owning the system includes knowing the boundary of its evidence:

Local and SQLite tests do not prove PostgreSQL concurrency, distributed availability, or production capacity.
The in-memory admission state does not provide fleet-wide enforcement.
Synthetic sources and simulation do not prove real-shopper relevance, conversion, revenue, retention, attribution, or causal uplift.
A generated-source receipt, including a PostgreSQL receipt, is not a PostgreSQL Qualification Claim by itself.
Snapshot candidate features are published, but the current Shopper interaction projection does not derive category/brand affinity maps. Concentration controls and affinity signals are separate.
Geographic publication currently returns at its 50,000-set cap instead of failing as proposed in the draft design; see the known limitations.
ANN and the two-tower option require separate relevance, memory, and latency qualification.
The published scale objectives require the unmodified qualification profile, approved PostgreSQL environment, and retained qualification evidence. They are not established by the focused suite.
Quality drift describes aggregate input/output change, not business outcome drift.
The original V1 design predates personalization, category/geographic Swimlanes, ANN, Polars, and the full simulation harness; their as-built contracts live in current code, migrations, tests, and dedicated docs rather than that original design record.

Qualification is driven by scripts/qualify.py, while the evidence meaning and limitations are specified in commerce-simulation-requirement-matrix.md and technical-implementation.md.

23. Suggested learning path¶

To move from user to owner, follow one request through each boundary:

Read the canonical language in CONTEXT.md.
Run the smoke Generated Relational Data Source and inspect its manifests.
Follow POST /v1/training-runs from api.py to TrainingRunRepository.create_with_outcome.
Follow TrainingWorker.run_once through SourceReadSession, generate_recommendations, and SnapshotRepository.publish.
Query one anchored and one global lane; inspect Recommendation Provenance and stale metadata.
Run the independent simulation and compare source deltas, checkpoint attribution, and report unavailable reasons.
Enable personalization only in a disposable local scope; append ordered interactions, replay an idempotency key, provoke a sequence conflict, then serve with a causal token.
Read the focused tests beside each component before modifying it.
End every change with the repository's narrow-to-wide verification ladder and a final diff review.

The shortest authoritative references after this guide are the architecture overview, technical implementation reference, approved specification, and testing guide.