Design: Commerce recommendation service

This document records the original design baseline. The current implementation has eleven trained strategies, category/geographic Swimlanes, default-off personalization and ANN, and a selectable Polars aggregate engine. Original six-strategy references describe that baseline, not the present registry. See engineering rationale for current decisions, tradeoffs, and known differences; this note does not retroactively approve newer proposals.

Control

Field	Value
Status	APPROVED
Design revision	0.3
Owner	Codex (design author); project technical owner not separately named
Reviewer / approval	Sponsor (user), approved 2026-09-21, design revision 0.3
Governing source	specs/commerce-recommendation-service.md, CRS-001 revision 0.14, approved 2026-09-21
Delivery unit	Greenfield V1 Training API, batch pipeline, snapshot store, and Serving API
Change class	Greenfield

1. Outcome and scope

Outcome: An internal caller can start a property/catalog-scoped batch run over directly streamed relational rows and later retrieve an atomically published, provenance-bearing recommendation snapshot within the approved service objectives.

In scope

• Python service bootstrap, versioned HTTP contracts, durable Training Run coordination, relational source adapters, six recommendation strategies, evaluation, atomic publication, retention, and serving.
• PostgreSQL as the production service-owned control and snapshot store; SQLite as a local and focused-test substitute.
• A single-node, out-of-core batch worker that can scale horizontally across independent property/catalog boundaries.
• Compatibility certification for PostgreSQL, SQLite, and Snowflake sources through one adapter contract.

Out of scope

• Source-system schema management, credentials UI, scheduling, shopper personalization, online learning, cross-property behavior, cross-Catalog results, recommendation rendering, and experiment assignment.
• A distributed compute engine in V1. The interfaces below permit replacing the batch executor if qualification evidence later requires it.
• Authentication and authorization, per CRS-001 DEC-3.

Must remain unchanged

• External systems continue to load source data and schedule runs.
• Serving never reads source interactions or computes recommendations synchronously.
• Raw source rows are neither staged nor persisted; only derived aggregates, metrics, and recommendation artifacts may use worker scratch or durable service storage.

2. Current system and constraints

Evidence	Current fact	Design consequence
main.py	PyCharm sample prints a greeting; there is no application entry point.	Replace it with package entry points; there is no behavioral compatibility surface.
pyproject.toml	Project declares Python >=3.9 and no dependencies.	Establish the runtime, dependency groups, packaging, and test commands explicitly.
.venv/pyvenv.cfg	Local environment is Python 3.9.6 and has no pip module.	Do not treat the checked local environment as a runnable baseline; bootstrap a Python 3.12 environment before the first red test.
specs/commerce-recommendation-service.md	CRS-001 revision 0.13 is approved and defines REQ-001 through REQ-026.	This design may choose implementation details but may not change observable behavior.
Repository root	No .git, CI definition, tests, migrations, containers, or repository-local AGENTS.md exist.	Add only the minimum project structure; CI/deployment integration remains an operator-owned follow-up.

• Dependency evidence: Python 3.9 reached end of life on 2025-10-31, so V1 targets Python 3.12 (Python version status). SQLAlchemy supports bounded yield_per/server-side result streaming and warns that materializing the result defeats it (SQLAlchemy streaming results). DuckDB supports streaming execution and out-of-core aggregation with bounded memory and scratch spill (DuckDB memory management). FastAPI recommends a separate worker system for heavy computation rather than in-process background tasks (FastAPI background tasks).
• Baseline behavior/evidence: Greenfield. .venv/bin/python main.py prints Hi, PyCharm; no test runner is installed and no automated baseline exists.

3. Proposed design

3.1 Decisions

ID	Decision	Rationale and evidence	Traces to
D-01	Use a Python 3.12 modular service with separate api and worker process entry points sharing application/domain modules.	Separates low-latency serving from CPU/I/O-heavy batch work and replaces the EOL repository runtime.	REQ-011, REQ-015, REQ-024
D-02	Use FastAPI/Pydantic for /v1 HTTP contracts and generated OpenAPI; execute no training work in the API process.	Typed response models validate and document the public seam; a durable worker survives caller disconnects.	REQ-011, REQ-012, REQ-015
D-03	Use a service-owned PostgreSQL database for durable coordination and snapshots, accessed with SQLAlchemy 2 and migrated by Alembic. SQLite supports local/focused tests, not the production concurrency or availability claim.	PostgreSQL transactions, partial unique indexes, row locking, and indexed JSONB rows make the lifecycle and atomic head swap explicit.	REQ-009, REQ-010, REQ-016 through REQ-019, REQ-022, REQ-024
D-04	Provision source adapters outside the API. Each adapter opens a consistent read session and returns bounded iterators from configured, parameterized, ordered queries through SQLAlchemy Core.	Preserves engine-neutral mappings and direct streaming. Result.all(), DataFrame construction over interaction rows, copy/export commands, and raw staging tables/files are forbidden.	REQ-002 through REQ-005, REQ-014, REQ-020, REQ-021
D-05	Require a source consistency token: one repeatable-read snapshot or engine time-travel point shared by Catalog, view, and purchase streams.	A common cutoff alone does not prevent late/backdated writes from changing a multi-query run. An adapter that cannot prove this contract is not certified.	REQ-009, REQ-014; NFR-003, RISK-03
D-06	Use one ephemeral DuckDB work database per run for derived support counts, pair counts, category pairs, daily popularity buckets, evaluation aggregates, and bounded top-N state. Delete it after success or failure.	Out-of-core aggregation bounds RAM without copying raw rows. DuckDB scratch spill contains derived keys/counts only.	REQ-006 through REQ-010, REQ-021, REQ-023 through REQ-025
D-07	Generate exact group-level co-occurrence counts from ordered streams; deduplicate Products within a group for pair evidence, preserve event/quantity multiplicity for popularity, and enforce configured positive distinct-Product limits before emitting any pairs, defaulting to 100 per session and 200 per Order.	Implements matrix co-occurrence semantics while preventing partial or biased truncation of oversized groups and permits deployment-specific tuning without code changes.	REQ-006, REQ-007, REQ-025; RULE-001, RULE-020 through RULE-022
D-08	Rank behavioral pairs with regularized cosine, fill candidates lexicographically by the approved fallback tiers, and store tier-specific provenance.	Normalization reduces popularity domination; tier-first filling prevents incomparable scores from different evidence types from being blended accidentally.	REQ-006, REQ-008, REQ-012; RULE-002 through RULE-005
D-09	Build metadata Similar Items as sparse, L2-normalized feature vectors and compute bounded top-N cosine products with scipy/sparse-dot-topn; do not introduce an embedding service in V1.	Deterministic structured/text metadata is reproducible and materially simpler than a network model dependency at 200k+ Products.	REQ-006, REQ-008, REQ-023; RISK-02
D-10	Use a chronological holdout, tune only an explicit recorded parameter grid, compare approved baselines, then rebuild the published candidates from the full cutoff window using the selected parameters.	Prevents temporal leakage while allowing the final snapshot to use all available historical evidence.	REQ-009, REQ-013; NFR-003, NFR-005
D-11	Coordinate jobs through durable training_run rows claimed by lease; enforce one active run with a database constraint and idempotency with a unique key.	A database-backed queue avoids another service while making conflict and crash recovery transactional.	REQ-015, REQ-016
D-12	Write immutable BUILDING snapshot rows and recommendation-set payloads in chunks, validate a six-strategy manifest, then atomically swap a small serving_head pointer and mark the run succeeded.	Chunk loading avoids a huge transaction; the invisible build state plus atomic head swap preserves all-or-nothing serving.	REQ-009, REQ-010, REQ-018
D-13	Store one pre-ranked JSONB payload per (snapshot, strategy, anchor) with at most 100 entries, rather than one row per candidate.	One indexed lookup and bounded decode supports the 90/150 ms serving target and reduces snapshot row count by roughly two orders of magnitude.	REQ-011, REQ-012, REQ-019, REQ-024
D-14	Run snapshot/run retention only after successful publication and never delete the serving head.	Makes cleanup recoverable and preserves stale serving when newer runs fail.	REQ-010, REQ-017, REQ-022
D-15	Emit structured logs and metrics keyed by run, property/catalog, source adapter, stage, cutoff, row counts, excluded groups, derived pair counts, duration, scratch high-water mark, fallback mix, and API latency.	Provides evidence for isolation, exclusions, the 12-hour objective, stale state, and failure diagnosis without raw events.	REQ-012, REQ-013, REQ-022, REQ-024, REQ-025
D-16	Qualify first on a 32-vCPU, 128-GiB RAM, 2-TB NVMe worker with a 10-Gbit/s source path; use bounded fetch/aggregate batches and fail explicitly on scratch or deadline exhaustion.	Establishes a repeatable initial measurement context. Capacity may scale upward without changing contracts.	REQ-023, REQ-024; NFR-006 through NFR-009
D-17	Treat the required purchases_query as Online Purchases and accept an optional offline_purchases_query. Stream the online query first and the offline query second inside the same consistent read session; tag rows with their Purchase Channel and prefix the internal Order grouping key by channel.	Preserves the existing online-purchase path, makes in-store input genuinely optional, avoids concurrent source cursors, and prevents matching cross-channel Order IDs from producing false pairs.	REQ-026; RULE-023, RULE-024

Consequential alternatives:

Alternative	Advantage	Rejection reason
Export source rows to Parquet/object storage, then train	Easy replay and distributed scanning	Directly violates REQ-021 and the approved no-extraction boundary.
FastAPI in-process background task	Minimal components	Worker loss follows API restarts and heavy work competes with serving; official guidance reserves it for smaller same-process tasks.
Spark-first batch engine	Straightforward scale-out and mature shuffle	Generic group/sort plans can spill canonical interaction rows, source certification becomes JDBC/cluster-specific, and operational cost is premature before the reference qualification.
One relational row per recommended candidate	Simple SQL filtering	Multiplies storage/index entries and query work; the contract always retrieves one bounded pre-ranked list.
Approximate neural/content embeddings	Potential cold-item quality gain	Adds nondeterminism, model distribution, and inference dependencies without local evidence that it beats sparse metadata cosine.

3.2 Change surface

Path / artifact	Add or change	Responsibility after change	Traces to
pyproject.toml	Replace starter metadata; require Python 3.12; define runtime, source extras, test, lint, and typing dependencies plus recommendations-api/recommendations-worker scripts.	Reproducible package and command boundary.	D-01, D-02, D-16
main.py	Remove starter behavior or retain only a compatibility message pointing to package entry points.	No application logic.	D-01
src/recommendations/config.py	Add deployment/data-source configuration models and validation.	Resolve stable data_source_id, service DB, worker budgets, query mappings, and property freshness.	D-04, D-16
src/recommendations/contracts.py	Add Pydantic request/response/error models and strategy enums.	Versioned API vocabulary independent of persistence classes.	D-02, D-13
src/recommendations/api.py	Add app factory and /v1 route handlers.	Validate calls, invoke application services, map domain errors, expose health/metrics.	D-02, D-11 through D-15
src/recommendations/domain.py	Add identities, run/snapshot states, recommendation entries, provenance, confidence, and invariants.	Pure product rules and deterministic ordering.	REQ-004 through REQ-008, REQ-017 through REQ-019
src/recommendations/source.py	Add SourceAdapter, SourceReadSession, canonical row types, and SQLAlchemy implementation.	Consistent, ordered, bounded source streams with validation and no raw staging.	D-04, D-05
src/recommendations/pipeline/cooccurrence.py	Add grouped stream reducer, limit handling, pair/support aggregation, regularized cosine, and category pairs.	FBT/Also Viewed primary evidence and exclusion metrics.	D-06 through D-08
src/recommendations/pipeline/content.py	Add deterministic metadata vectorization and sparse top-N similarity.	Similar Items and Also Viewed content fallback.	D-09
src/recommendations/pipeline/popularity.py	Add daily buckets, decay, momentum, and category/property rankings.	Best Sellers, Most Viewed, Trending, and popularity fallback tiers.	D-07, D-08
src/recommendations/pipeline/evaluation.py	Add chronological split, metrics, baselines, support cohorts, and parameter selection.	Reproducible evaluation evidence.	D-10
src/recommendations/pipeline/run.py	Add end-to-end stage orchestration and six-strategy manifest.	One cutoff, one configuration, all-or-nothing run outcome.	D-05, D-06, D-10, D-12
src/recommendations/workstore.py	Add ephemeral DuckDB schema, batched derived upserts, top-N scans, cleanup, and resource limits.	Larger-than-memory derived computation without raw staging.	D-06, D-16
src/recommendations/storage.py	Add SQLAlchemy service-store repositories and atomic publication/retention operations.	Durable runs, snapshots, head lookup, and cleanup.	D-03, D-11 through D-14
src/recommendations/worker.py	Add lease-based claimant, heartbeat, run execution, crash recovery, and terminal error mapping.	Durable asynchronous execution outside API processes.	D-01, D-11, D-15
migrations/versions/*	Add authoritative Alembic migrations for the schema below.	Reproducible production schema evolution and downgrade.	D-03, D-12 through D-14
tests/unit/*	Add pure rule, algorithm, ordering, fallback, and metric cases.	Fast deterministic TDD seam.	D-07 through D-10
tests/contract/*	Add HTTP/OpenAPI and source adapter contract cases.	Stable consumer/provider behavior.	D-02, D-04, D-13
tests/integration/*	Add PostgreSQL lifecycle/publication/lease tests and SQLite source certification.	Transactional and real-boundary evidence.	D-03, D-05, D-11 through D-14
tests/performance/*	Add generated workload profiles and API/batch measurement harness.	Retained evidence for NFR-006 through NFR-009.	D-16

No generated source is planned. OpenAPI JSON is runtime-derived from contracts.py; CI captures it as a comparison artifact with python -m scripts.export_openapi, but the captured file is not authoritative.

3.3 Interfaces and invariants

HTTP API

All responses are JSON. Identifiers are non-empty UTF-8 strings up to 255 bytes; the service compares them bytewise after Unicode NFC normalization. Times are UTC RFC 3339. Scores are finite JSON numbers.

Operation	Request	Success	Material errors
POST /v1/training-runs	Header Idempotency-Key (1–128 chars); body {data_source_id, tracking_id, catalog_id}	202 with {run_id,status,scope,requested_at}; same key returns the same body/run	404 data_source_not_found; 409 training_run_active with active run; 422 invalid_request
GET /v1/training-runs/{run_id}	Run ID	200 with state, timestamps, cutoff, metrics summary, failure category, and snapshot ID when applicable	404 training_run_not_found
GET /v1/catalogs/{catalog_id}/items/{product_id}/recommendations/{strategy}	data_source_id, tracking_id, limit; strategy is frequently-bought-together, also-viewed, or similar-items	200 RecommendationResponse	404 snapshot_not_found or item_not_found; 422 invalid_limit/scope/strategy; 503 snapshot_incomplete only for detected corruption
GET /v1/catalogs/{catalog_id}/recommendations/{strategy}	data_source_id, tracking_id, limit; strategy is best-sellers, most-viewed, or trending	200 RecommendationResponse	Same snapshot/validation mappings as anchored retrieval

RecommendationResponse contains strategy, optional anchor_product_id, reason (null or insufficient_evidence), snapshot_id, generated_at, data_cutoff, is_stale, and items. Each item contains product_id, one-based rank, score, confidence_tier (high|medium|low), and provenance (source_tier, evidence_type, support, optional fallback_reason). It never contains Tracking ID from another scope, source credentials, Order ID, Session ID, or raw events.

The database-stored order is authoritative. Serving slices the first limit entries and does not rescore. Entries are score-descending within one evidence tier; candidate selection fills earlier approved tiers completely before later tiers, with Product ID ascending for equal scores. Anchor/self, unavailable, duplicate, and cross-boundary candidates are removed before rank assignment.

Source adapter

SourceAdapter.open(scope, history_start, cutoff) -> SourceReadSession
SourceReadSession.consistency_token -> opaque string
SourceReadSession.stream_catalog(fetch_size) -> Iterator[CatalogRow]
SourceReadSession.stream_views(fetch_size) -> Iterator[ViewRow]
SourceReadSession.stream_purchases(fetch_size) -> Iterator[PurchaseRow]

The deployment configuration names environment-held connection URLs, three required pre-approved SQL statements or relation mappings, and one optional mapping. purchases_query reads Online Purchases; offline_purchases_query reads Offline Purchases when the merchandiser supplies them. Every configured statement binds tracking_id, catalog_id, history_start, and cutoff; adapters may bind a consistency token. Canonical rows are:

Stream	Required fields	Optional fields	Required order
Catalog	catalog_id, product_id, is_eligible	category_id, brand, price, created_at, title, description, additional mapped metadata	product_id
Views	tracking_id, catalog_id, product_id, session_id, event_at	none	session_id, event_at, product_id
Online Purchases	tracking_id, catalog_id, product_id, order_id, event_at, quantity	none	order_id, event_at, product_id
Offline Purchases	same as Online Purchases when configured	entire stream	order_id, event_at, product_id

The adapter uses Core rows, yield_per, and iterator partitions. It tags purchase rows as online or offline from their configured query, without requiring a channel column from the source. Online and Offline Order IDs occupy separate internal grouping namespaces. It must never call all(), build an interaction DataFrame, issue COPY/unload/export, create a source-side staging relation, or write canonical rows to worker disk. Every row is boundary-validated while consumed. Any ordering break, scope mismatch, missing required value, non-positive quantity, invalid time, lost consistency token, or premature cursor end fails the run as source_invalid or source_incomplete.

Service-owned schema

Table	Key and material columns	Invariants
training_run	run_id; scope; idempotency_key; request_hash; state/times/cutoff; config/model versions; lease owner/expiry; failure category; metrics JSON; snapshot ID	Unique (scope,idempotency_key); partial unique active scope for pending|running; terminal rows immutable except retention deletion
recommendation_snapshot	snapshot_id; scope; run ID; state building|available; generated/cutoff/freshness; config/model versions; strategy manifest JSON	One snapshot per successful run; only complete six-strategy manifests may become available
recommendation_set	(snapshot_id,strategy,anchor_key); outcome; entries JSONB	anchor_key='' only for global strategies; payload has 0–100 pre-ranked entries and no duplicate/self/ineligible candidate
serving_head	scope primary key; snapshot ID	References one available snapshot of the identical scope
failed_run_diagnostic	run ID; category; redacted detail; expiry	No raw rows or credentials; default expiry 30 days

training_run.metrics retains evaluation metrics, counts, durations, and exclusion aggregates for 90 days. Snapshot retention keeps the five newest available snapshots per scope. A retention transaction locks the scope, re-reads the head, and deletes only non-head snapshots beyond five.

Training and publication states

pending --lease claim--> running
running --six valid strategy outcomes + atomic head swap--> succeeded
running --validation/read/compute/evaluation/publication failure--> failed

An expired worker lease permits another worker to restart the same run ID from the beginning without changing its visible running state. At most one recovery attempt is made; a second lost lease fails the run as worker_lost. Recovery deletes only that run's ephemeral work database and invisible building snapshot rows. It never mutates serving_head until final publication. This is crash recovery, not caller-visible cancellation or a retry of a terminal run.

Algorithms

For strategy s, let C_s(i) be the number of eligible groups containing Product i, and C_s(i,j) the number containing both distinct Products. Each Product contributes at most once to pair evidence per group. The primary pair score is:

cosine(i,j) = C(i,j) / sqrt(C(i) * C(j))
score(i,j)  = cosine(i,j) * C(i,j) / (C(i,j) + shrinkage)

Pairs below selected min_support are ineligible. min_support and shrinkage come from a bounded deployment grid, are selected independently for FBT and Also Viewed by chronological NDCG@20, and are recorded in the snapshot. Known-literal tests use fixed parameters rather than calling the production scorer as their oracle.

• FBT: group by Order; pair evidence is unique Product presence, while Best Sellers uses summed positive quantity. Fallback fill order is primary pair, category-pair co-purchase plus within-category Best Seller, supplied complement rule plus Best Seller, category Best Seller, property Best Seller. Metadata cosine is never an FBT tier.
• Also Viewed: group by Browsing Session; repeated views count once for pair evidence but every valid view counts for popularity. Fallback fill order is primary pair, metadata Similar Items, category Most Viewed, property Most Viewed.
• Similar Items: concatenate weighted sparse feature blocks: category and brand one-hot, log-price bucket within category, and word/character TF-IDF from title/description. L2-normalize each block, apply recorded weights, normalize the final vector, and retain bounded cosine top-N. An anchor without sufficient mapped metadata produces insufficient_evidence; no popularity tier is relabeled as similarity.
• Best Sellers / Most Viewed: sum weight * exp(-ln(2) * age_days / half_life_days), where weight is purchase quantity or one view. Half-life is selected from a recorded grid.
• Trending: compare a configured recent window with the immediately preceding baseline window: log2((recent_rate + alpha)/(baseline_rate + alpha)) * log1p(recent_count). Counts combine views and quantity-weighted purchases using recorded weights. Non-positive momentum is ineligible.

Oversized groups are detected only after the whole ordered group has been consumed. Their buffered unique Product set is bounded at limit + 1; after overflow, the reducer stops retaining additional unique IDs, continues counting rows/popularity, emits zero pairs/category pairs, and increments separate group/row exclusion metrics. Thus no partial group evidence is possible and memory remains bounded.

For every anchored strategy, candidate generation requests more than 100 candidates per tier to allow eligibility/self/dedup filtering, then retains exactly the first 100 or fewer. Confidence is high for primary behavioral evidence meeting the selected high-support threshold, medium for other primary/category/content/compatibility evidence, and low for category/property popularity. Provenance records the actual tier and support; confidence never changes the approved tier order.

Evaluation

• Assign an entire Order or Browsing Session to the holdout when its maximum event time is in the configured final holdout interval; do not split a group across train and holdout.
• Select parameters on pre-holdout derived counts and binary holdout neighbor sets. Report Recall@K and NDCG@K for K 10,20,100, catalog coverage, top-1%-item exposure share as popularity concentration, fallback rate, and warm/sparse/strict-cold cohorts.
• Compare raw pair count, regularized cosine, and time-decayed popularity. Select highest NDCG@20; break exact ties by Recall@20, then coverage, then the lower-complexity parameter tuple.
• After selection, merge train and holdout derived aggregates to produce the cutoff snapshot. The report retains both evaluation-period and final-build parameter identities.
• A metric computation failure fails the run. A valid empty cohort is reported as absent and does not fabricate a zero-quality claim.

3.4 Runtime and data flow

Caller -> API -> PostgreSQL training_run(pending)
                    |
                 Worker lease
                    |
        configured relational source
          | catalog/views/online purchases/[optional] offline purchases
          v (bounded ordered iterators; no raw staging)
        validators/group reducers
          v
       ephemeral derived DuckDB -> evaluation -> six ranked strategy sets
                    |
            PostgreSQL BUILDING snapshot
                    |
          atomic manifest + serving_head swap
                    v
Serving caller -> API -> serving_head -> one recommendation_set payload

1. POST validates the configured source identity and inserts a pending run transactionally. A disconnect after 202 has no effect.
2. A worker claims the run, fixes cutoff=claim time, derives history_start=cutoff-2 years unless a property requests a longer window, opens one consistent source snapshot, and streams Catalog, views, required Online Purchases, and configured optional Offline Purchases.
3. Catalog eligibility/metadata stays in a bounded in-memory/sparse representation. View and purchase reducers update popularity counters immediately, but emit pair/category deltas only after a group closes within its approved size.
4. Batched derived deltas are merged into the run-scoped DuckDB work database. No canonical row or complete source group is written there.
5. Evaluation selects parameters, candidate generators construct six complete strategy manifests, and the publisher writes invisible snapshot payloads in chunks.
6. One transaction verifies the manifest, marks the snapshot available, swaps serving_head, and marks the run succeeded. Cleanup then removes scratch and applies retention.
7. Any failure closes source cursors, removes scratch/building rows, records a bounded diagnostic, marks the run failed, and leaves the previous head unchanged.

3.5 Impact screen

Concern	Applies?	Design / verification
Dependencies and external services	yes	FastAPI, Pydantic, SQLAlchemy/Alembic, PostgreSQL driver, source-driver extras, DuckDB, NumPy/SciPy/scikit-learn/sparse-dot-topn; lock resolved versions and certify each source extra.
Configuration, secrets, environments	yes	External config references connection URL environment variables; startup rejects missing mappings/budgets. Credentials never enter API/run/snapshot records.
Data model and migration	yes	Alembic creates/drops service-owned tables and indexes; no source migration or raw backfill.
API and backward compatibility	yes	New /v1 contract; OpenAPI snapshot comparison prevents accidental breaking changes.
Security and privacy	limited	No auth by approved scope. Bound parameters and response models remain correctness/data-isolation controls; no raw rows in durable or API data.
Reliability and concurrency	yes	Unique active boundary, idempotency constraint, worker lease, invisible builds, atomic head swap, stale-head preservation, and failure injection.
Performance and resources	yes	Fixed API SLOs; 12-hour qualification; bounded fetch/group/top-N state; DuckDB memory/scratch ceilings; production-equivalent benchmark.
Observability and operations	yes	Structured run/stage/exclusion/resource/evaluation/API metrics and failure categories; no raw payload logging.
Deployment and rollback	yes	Migrate DB, deploy worker, then API; rollback API/worker before migration downgrade; old head remains serveable.
Accessibility and interaction	no	Machine-only internal JSON APIs; no human interface is introduced.

3.6 Performance and resource controls

• API and worker are separately deployable. Serving starts with at least two API replicas; worker concurrency is configurable globally but scope uniqueness remains database-enforced.
• Source fetch_size defaults to 10,000 and is adapter-certified. In-memory pair delta batches default to 250,000 derived pairs. Both are configuration, recorded per run, and cannot change semantics.
• DuckDB receives explicit memory_limit=96GB, threads=32, a run-owned NVMe temp_directory, and max_temp_directory_size=1.5TB on the reference worker. Crossing a configured hard limit fails as resource_exhausted; it never falls back to raw staging.
• The qualification generator contains 200,000 Products, 100 million views, 100 million purchase rows, a two-year span, median/p95/max distinct sizes of 3/20/100 for sessions and 2/10/200 for Orders, 5% repeated views, 5% repeated purchase lines, and explicit oversized groups for exclusion accounting. The shape is a measurement fixture, not a new input limit.
• API latency is measured at the service boundary against a populated 200,000-Product snapshot, with 90% anchored hits, 5% valid empty sets, and 5% structured misses. Batch time includes source-read latency over the 10-Gbit/s reference path.

4. Test design

4.1 Seams and strategy

Seam	Why behavior is observable here	Level / size	Harness	Focused command after bootstrap
Pure domain/pipeline public functions	Exact grouping, scoring, fallback, metrics, ordering	Unit / small	pytest, literal rows, fake clock, temporary DuckDB	python -m pytest tests/unit -q
FastAPI app factory	Versioned request/response/error semantics without a network	Contract / small	TestClient, isolated repository fake or SQLite	python -m pytest tests/contract/test_api.py -q
SourceAdapter compatibility suite	Real cursor streaming, ordering, cutoff, and scope validation	Contract/integration / medium	SQLite fixture always; PostgreSQL/Snowflake certification URLs by marker	python -m pytest tests/contract/test_source_adapter.py -q
PostgreSQL repositories + worker	Constraints, leases, chunk publication, atomic head, retention	Integration / medium	Disposable PostgreSQL supplied by TEST_CONTROL_DATABASE_URL; barriers, not sleeps	python -m pytest -m postgres -q
Full app + worker + source	Critical asynchronous journey and recovery	End-to-end / medium	Disposable PostgreSQL and SQLite source, controlled worker tick	python -m pytest tests/integration/test_end_to_end.py -q
Qualification harness	Capacity, duration, latency, resource and availability evidence	Performance / large	Dedicated reference environment and deterministic synthetic seed	python -m pytest -m performance --benchmark-artifact=artifacts/qualification.json

Shared controls: UTC fake clock; fixed float tolerances and parameter grids; deterministic Product IDs; seeded workload generation; run-owned temp directories; no live production data; database transactions isolated per case; concurrency coordinated by barriers; source failures injected at iterator boundaries. Interaction assertions are limited to forbidden source export/staging and atomic publication boundaries because those interactions are themselves required behavior.

4.2 Verification cases

ID / exact test name	Covers	Level, seam, location	Given / When	Then / independent oracle	Data, doubles, controls	Expected pre-change signal
V-01 test_submit_returns_durable_pending_run	REQ-001,015	contract; API; tests/contract/test_api.py	Valid configured scope; POST with key	202, stable run ID/pending row; oracle CRS EX-007	fake config, frozen clock	RED: app/import absent
V-02 test_unknown_source_rejects_without_run	REQ-002,020	contract; API	Missing data_source_id; POST	404 code and zero runs; oracle EX-017	repository fake	RED: app absent
V-03 test_same_idempotency_key_returns_existing_run	REQ-016	PostgreSQL integration	One pending run; repeat same key/scope	Same ID, one row; oracle EX-008	real unique constraint	RED: schema absent
V-04 test_different_request_reports_active_run	REQ-016	PostgreSQL integration	Active run; second key same scope	409 names active ID, no queue row; oracle EX-009	two transactions + barrier	RED: schema absent
V-05 test_stream_rejects_cross_scope_row	REQ-003 through 005	integration; source/pipeline	Adapter yields one wrong Tracking/Catalog ID	Run fails source_invalid, no snapshot; oracle EX-001	iterator fake, literal IDs	RED: source seam absent
V-06 test_fbt_counts_only_unique_products_in_same_order	REQ-006,007	unit; co-occurrence	A/B same Order, duplicate A line, C other Order	C(A,B)=1, C(A,C)=0; oracle EX-002	literal rows	RED: reducer absent
V-07 test_also_viewed_counts_only_same_session	REQ-006,007	unit; co-occurrence	A/B same session, C different, repeated A view	C(A,B)=1, repeated view does not add pair	literal rows	RED: reducer absent
V-08 test_group_limits_are_inclusive	REQ-025	unit; group reducer	100-item session and 200-item Order	Both emit n(n-1)/2 eligible pair deltas; oracle EX-022 literal formula	generated sequential IDs	RED: limits absent
V-09 test_oversized_groups_emit_no_pairs_but_keep_popularity	REQ-025	unit; group reducer	101-item session, 201-item Order	Zero pair/category deltas; all rows/quantities counted; exact exclusion metrics; oracle EX-023	bounded iterator; inspect public result	RED: exclusion absent
V-10 test_regularized_cosine_matches_literal_matrix	REQ-006,008	unit; scorer	Ci=4,Cj=9,Cij=3,shrink=3	score 0.25; oracle published equation arithmetic	fixed Decimal tolerance	RED: scorer absent
V-11 test_also_viewed_fallback_is_tier_ordered_and_deduplicated	REQ-008	unit; ranker	Primary, content, category, property candidates overlap	Earlier tier wins; no duplicate/self; oracle RULE-002	literal scored candidates	RED: ranker absent
V-12 test_fbt_never_uses_similarity_as_complement	REQ-008	unit; ranker	Only a similar substitute plus popularity exists	Substitute not labeled/used before approved purchase tiers; oracle EX-004	literal provenance	RED: ranker absent
V-13 test_similar_items_returns_metadata_cosine_or_empty	REQ-006,008	unit; content	Two shared-metadata items and one metadata-empty anchor	Known cosine neighbor; empty anchor => insufficient; oracle RULE-005	fixed vectorizer vocabulary	RED: content absent
V-14 test_popularity_decay_and_trending_match_literals	REQ-006	unit; popularity	Fixed timestamps/counts/half-life/alpha	Exact exponential weights and momentum order; oracle equations in D-08	frozen UTC clock	RED: calculators absent
V-15 test_temporal_holdout_keeps_groups_whole_and_reports_cohorts	REQ-013	unit; evaluation	Group spans boundary; warm/sparse/cold items	Group assigned by max time; literal Recall/NDCG and cohort keys	literal rankings/ground truth	RED: evaluator absent
V-16 test_incomplete_source_stream_fails_and_preserves_head	REQ-010,018,021	end-to-end	Existing head; cursor fails mid-stream	Run failed, head/payload unchanged, no raw artifact; oracle EX-005	faulting iterator, temp-dir inventory	RED: orchestration absent
V-17 test_strategy_failure_never_publishes_mixed_snapshot	REQ-010,018	integration; publisher	Five strategy sets, sixth fails	No available new snapshot/head change; oracle EX-014	failure injection before manifest	RED: publisher absent
V-18 test_insufficient_evidence_is_publishable_empty_strategy	REQ-017,018	end-to-end	Valid sparse data lacks FBT pairs	Snapshot publishes six manifests; FBT empty/reason; oracle EX-013	SQLite source	RED: pipeline absent
V-19 test_atomic_head_swap_exposes_only_complete_snapshot	REQ-009,010,018	PostgreSQL integration	Reader loops while chunk build and final swap occur	Every read sees old complete or new complete snapshot, never building/mix	barriers around commit	RED: schema absent
V-20 test_serving_distinguishes_missing_empty_and_stale	REQ-017	contract; API	Three database states	404 snapshot_not_found; 200 empty reason; stale 200 metadata; oracle EX-010–012	frozen clock	RED: routes absent
V-21 test_serving_limit_boundaries_and_stable_ties	REQ-019	contract; API	101 entries with equal-score A/B	default 20; 1/100 valid; 0/101 rejected; A before B; oracle EX-015/016	literal stored payload	RED: routes absent
V-22 test_serving_filters_self_ineligible_and_cross_scope	REQ-005,012	integration; serving	Corrupt candidate fixture includes forbidden entries	Public response contains none and reports corruption rather than leaking	adversarial payload	RED: serving absent
V-23 test_sqlite_adapter_streams_in_order_with_bounded_fetches	REQ-014,021	source contract	Large SQLite fixture and configured query	Ordered canonical iterator; bounded observed batch; no export/stage	instrumented DBAPI cursor	RED: adapter absent
V-24 test_adapter_loses_consistency_token_and_fails_run	REQ-009,014	source contract	Token changes between streams	source_incomplete, no publication; oracle D-05	fake adapter tokens	RED: contract absent
V-25 test_repository_contains_no_raw_staging_path	REQ-021	static/contract	Scan config/schema/export calls plus run fixture	Only approved derived tables/files; no Parquet/COPY/raw table; oracle EX-018	allowlist of migration/workstore names	RED: project structure absent, then sensitive to forbidden addition
V-26 test_retention_keeps_five_snapshots_and_never_deletes_head	REQ-022	PostgreSQL integration	Six snapshots and expired run/diagnostic rows	Five newest/head remain; 90/30-day records obey boundaries; oracle EX-019	frozen clock, real FK	RED: retention absent
V-27 test_worker_reclaims_once_then_fails_second_lost_lease	REQ-015,016	PostgreSQL integration	Expired running lease twice	Same run restarts once; second loss terminal worker_lost; head unchanged	fake monotonic clock, barriers	RED: worker absent
V-28 test_identical_inputs_produce_equivalent_snapshot	REQ-009,013	end-to-end	Two runs, identical streams/cutoff/config/version	Byte-equivalent ordered payloads and metrics within declared float encoding	fixed seed/clock	RED: pipeline absent
V-29 test_capacity_run_completes_without_raw_staging	REQ-023,025	performance; full pipeline	Approved synthetic qualification profile	Success within 12h, exclusion metrics exact, no raw artifacts, resource ceilings retained	reference worker, seeded generator	RED: harness/system absent
V-30 test_api_latency_objectives	REQ-024	performance; HTTP	Representative populated workload	Training p95 ≤500ms; serving p95 ≤90ms/p99 ≤150ms	warm-up, fixed duration/concurrency, raw percentile artifact	RED: API absent
V-31 monthly_availability_calculation_and_operational_evidence	REQ-024	analysis/rehearsal	Calendar-month request/SLO telemetry	Serving ≥99.9%, Training ≥99.5%; retained report uses defined denominator	operations-owned telemetry query	RED: signals/runbook absent
V-32 test_optional_offline_purchases_are_combined_without_merging_orders	REQ-026	source contract/unit reducer	Online and Offline streams contain different Items under the same source Order ID; repeat without Offline mapping	Both channel-local pairs contribute when configured; no cross-channel pairs; online-only configuration remains valid	SQLite source, literal rows/channels	RED: Offline mapping required or channels merged

V-08 intentionally verifies the exact quadratic pair count only at the approved boundary in the focused unit reducer; V-29 uses representative distributions and is the performance oracle. This separates semantic correctness from workload qualification.

4.3 Traceability

Requirement / invariant / risk	Design decisions	Verification IDs	Coverage
REQ-001, REQ-002, REQ-015, REQ-020	D-02, D-04, D-11	V-01, V-02	covered
REQ-003 through REQ-005	D-04, D-05, D-07, D-13	V-05, V-22, V-23	covered
REQ-006, REQ-007	D-06 through D-10	V-06, V-07, V-10, V-13, V-14	covered
REQ-008	D-08, D-09	V-11 through V-13	covered
REQ-009, REQ-013	D-05, D-10, D-12, D-15	V-15, V-19, V-24, V-28	covered
REQ-010, REQ-018	D-11, D-12	V-16 through V-19, V-27	covered
REQ-011, REQ-012	D-02, D-13	V-20 through V-22	covered
REQ-014	D-04, D-05	V-23, V-24; PostgreSQL/Snowflake certifications use same suite	covered
REQ-016	D-03, D-11	V-03, V-04, V-27	covered
REQ-017	D-13, D-14	V-18, V-20	covered
REQ-019	D-13	V-21	covered
REQ-021	D-04, D-06	V-16, V-23, V-25, V-29	covered
REQ-022	D-14, D-15	V-26	covered
REQ-023	D-06, D-09, D-16	V-29	covered
REQ-024	D-01, D-02, D-03, D-13, D-16	V-29 through V-31	covered
REQ-025	D-07, D-15, D-16	V-08, V-09, V-29	covered
REQ-026	D-04, D-05, D-17	V-23, V-32	covered
NFR-001 through NFR-005	D-04, D-05, D-10 through D-15	V-05, V-15 through V-19, V-22, V-24, V-28	covered
NFR-006 through NFR-009	D-13, D-16	V-29 through V-31	covered
RISK-01 pair explosion	D-06, D-07, D-16	V-08, V-09, V-29	covered
RISK-02 fallback distortion	D-08 through D-10	V-11 through V-15	covered
RISK-03 dialect consistency	D-04, D-05	V-23, V-24 and adapter certification	covered

Every decision D-01 through D-17 and every changed surface traces to at least one approved requirement or named risk. No verification case introduces an unapproved behavior.

5. TDD delivery plan

Bootstrap is a prerequisite, not a horizontal implementation phase: create Python 3.12 packaging with pytest and an empty package, then execute each case below as its own red-green-refactor cycle.

Slice	Test order, one cycle at a time	Minimum capability after green	Expected surfaces	Focused completion signal
S-01 Walking skeleton	V-01, then V-02	API can durably accept/inspect a configured run without executing it	pyproject.toml, contracts, API, config, minimal storage	python -m pytest tests/contract/test_api.py -q
S-02 Durable concurrency	V-03, then V-04	Idempotency and one-active-scope behavior are database-enforced	migration, storage	python -m pytest -m postgres -k 'idempotency or active_run' -q
S-03 Source boundary	V-23, then V-05, then V-24, then V-32	Certified direct streams with scope/order/consistency failure semantics and optional channel-safe Offline Purchases	source, config	python -m pytest tests/contract/test_source_adapter.py tests/unit/test_config.py -q
S-04 Co-occurrence core	V-06, then V-07, then V-10	Exact FBT/Also Viewed counts and normalized score	cooccurrence, workstore	python -m pytest tests/unit/test_cooccurrence.py -q
S-05 Resource guard	V-08, then V-09	Inclusive group limits, no partial pairs, popularity preservation, metrics	cooccurrence, popularity	python -m pytest tests/unit/test_group_limits.py -q
S-06 Content and fallback	V-13, then V-11, then V-12	Similar Items and approved tiered fallback with provenance	content, domain/ranker	python -m pytest tests/unit/test_fallbacks.py tests/unit/test_content.py -q
S-07 Popularity portfolio	V-14	Best Sellers, Most Viewed, and Trending score deterministically	popularity	python -m pytest tests/unit/test_popularity.py -q
S-08 Evaluation	V-15	Chronological metrics, cohorts, baselines, and selection	evaluation	python -m pytest tests/unit/test_evaluation.py -q
S-09 Failure-safe pipeline	V-16, then V-18, then V-17	End-to-end run can fail safely or publish valid empty outcomes	pipeline run, worker, publisher	python -m pytest tests/integration/test_end_to_end.py -q
S-10 Atomic publication	V-19	Chunk builds remain invisible and head swap is atomic	storage, migration	python -m pytest -m postgres -k atomic_head -q
S-11 Serving	V-20, then V-21, then V-22	Complete retrieval contract, bounds, ordering, freshness, isolation	API, contracts, storage	python -m pytest tests/contract/test_api.py -k serving -q
S-12 Recovery and lifecycle	V-27, then V-26	Lease recovery and safe retention	worker, storage	python -m pytest -m postgres -k 'lease or retention' -q
S-13 Reproducibility/no staging	V-25, then V-28	Static and dynamic evidence for no raw staging and deterministic output	whole package/tests	python -m pytest -k 'raw_staging or equivalent_snapshot' -q
S-14 Capacity	V-29	Qualification-scale batch meets time/resource/streaming obligations	performance harness plus tuned implementation	performance command and retained JSON artifact
S-15 API objectives	V-30, then V-31	Latency and availability evidence is operationally reproducible	performance harness, metrics/runbook	performance command plus monthly SLO report

• Baseline command: None exists. Current observed baseline is Python 3.9.6, no pip/pytest, and no tests/ directory; all V-cases are new behavior and expected RED after bootstrap.
• Final required gates: python -m pytest tests/unit tests/contract -q; PostgreSQL integration command; each declared source adapter certification; python -m pytest tests/integration -q; static type/lint commands defined in pyproject.toml; V-29/V-30 qualification artifacts; V-31 operational rehearsal.
• Blocking order: S-01 precedes all HTTP work; S-02/S-03 precede worker orchestration; S-04–S-08 precede publication; S-09 precedes serving; performance tuning begins only after functional oracles are green. If V-29 shows the single-node design cannot meet 12 hours without raw staging, stop and revise this design rather than weaken CRS-001.

6. Delivery and operations

• Migration/deployment: Back up the service-owned PostgreSQL database; apply Alembic schema; deploy workers with polling disabled; deploy API and smoke V-01/V-02/V-20; enable workers; run a small property; then run qualification. Source databases receive no migration.
• Mixed version: A snapshot stores contract, model, config, and payload schema versions. New API code must read the previous payload schema during a rolling deploy. Workers only publish the current schema. Remove previous readers one release after all retained snapshots have advanced.
• Telemetry: Counters for submissions/conflicts/outcomes/failure categories, gauges for pending/running age and scratch high-water, histograms for stages and API latency, snapshot age, strategy/fallback/exclusion distributions. Alerts: serving SLO burn, oldest pending/running age, run over 10 hours, no fresh head at threshold, repeated adapter failure, scratch over 80%.
• Rollback: Disable worker claims, roll API/worker back, and keep serving_head on the last compatible snapshot. A failed new run/build needs no data rollback. Downgrade the migration only when no retained row uses the new schema; otherwise retain the additive schema.
• Cleanup: Run-owned scratch is deleted on success/failure/recovery startup. Orphan building snapshots older than the maximum run deadline plus one hour are safe to delete only when not referenced by a live run or head.
• Documentation: Add operator configuration, adapter certification, local setup, run failure categories, qualification procedure, and retention/rollback instructions to README.md during implementation.

7. Risks and resolved decisions

Risk / trigger	Prevention or detection	Mitigation / fallback	Owner
RISK-01 Derived pair cardinality exhausts time or scratch despite group caps	Pair-count/scratch metrics, 10-hour warning, V-29 profile	Tune derived batching/indexes or scale the worker; if 12 hours still fails, return to design for a distributed derived-aggregate executor—never raw staging	Engineering
RISK-02 Metadata or popularity fallback harms intent	Provenance, tier metrics, cohort evaluation, V-11–V-15	Tune/disable a failing fallback tier per property through approved configuration and republish; do not relabel evidence	Product/ML
RISK-03 A dialect cannot maintain a stable multi-stream snapshot	Compatibility token contract and V-24	Do not certify that adapter; add a source-specific consistent-read implementation	Data Engineering
RISK-04 PostgreSQL JSONB payload size or decode time misses serving SLO	V-30 payload/latency telemetry	Compress only after measuring; add an in-memory read-through cache keyed by immutable snapshot/set if required, preserving PostgreSQL authority	Engineering
RISK-05 Long source transactions affect merchant databases	Source duration/query-plan telemetry and adapter readiness review	Prefer engine time travel/read replicas; lower property concurrency; fail certification when safe consistency cannot be provided	Data Engineering/operator
RISK-06 Sparse content multiplication exceeds memory	Bounded top-N multiplication, block/resource telemetry	Reduce block size or scale worker; replace only the ContentCandidateGenerator behind its contract if qualification fails	ML Engineering

• Resolved assumptions: Python 3.12 replaces the EOL starter runtime; production coordination uses PostgreSQL while source compatibility remains adapter-based; configurable group limits and oversized behavior are approved in CRS-001 revision 0.14; the reference resource/workload shape is a technical qualification context and not a new product limit.
• Open questions: None before review.

8. Readiness

• [x] Authority, revision, scope, non-goals, and unchanged behavior are explicit.
• [x] Repository evidence supports every material current-state claim.
• [x] Decisions, interfaces, flows, errors, and applicable impacts are concrete.
• [x] Traceability is complete in both directions.
• [x] Every automated case has an independent oracle and expected pre-change signal.
• [x] Test data, boundary doubles, deterministic controls, environments, and commands are known.
• [x] TDD slices are vertical, dependency-ordered, and one-case-at-a-time.
• [x] Applicable migration, compatibility, rollout, observability, and rollback are safe.
• [x] Risks are owned; no material question remains.
• [x] The design introduces no requirement and requires no material interpretation.

Author gate: APPROVED. Sponsor (user) approved design revision 0.1 on 2026-08-05; implementation may proceed against this exact baseline.