Engineering rationale and current design

This is the explanation of the current system for an engineer who wants to own it. It complements the repository walkthrough and the technical reference, rather than replacing the behavioral spec. The source audit for this revision used commit 2d2907e on 2026-09-25. Links name modules and symbols rather than frozen line numbers so they remain useful as code moves.

Recorded rationale below refers to a reason written in a repository design or specification. Implementation inference explains consequences of the code; it does not claim to reconstruct an author's private reasoning. A test link identifies evidence to inspect, not a claim that that test was executed during this documentation revision. Existing drafts remain drafts.

Read in order for the system model, or jump to decisions, training, publication, serving, ANN, costs, or limitations. The companion implementation workflow explains how to make changes.

1. Concepts that must remain separate

The primary unit is a Commerce Scope, the triple (data_source_id, tracking_id, catalog_id). SKU-1 in two Catalogs is two different Items. Two data sources can legitimately reuse the same Tracking ID. A city name is neither a global identity nor an authorization boundary. Scope is therefore carried into configuration, source validation, database keys, profile keys, and lookup methods. Adding it only to the HTTP route would leave internal and background paths unsafe.

Concept	Owns or means	Does not mean
Recommendation Strategy	Product intent and allowed evidence order	A numerical algorithm or a UI row
Recommendation Set	One bounded ordered result for a strategy/anchor/variation	The entire trained generation
Recommendation Snapshot	Immutable published generation with sets and companion artifacts	A live view of the merchant database
Serving head	Which generation a scope currently serves	A mutable model being trained in place
Swimlane	Ordered accumulation across configured strategy/geography steps	A weighted blend of scores across strategies
Shopper Profile	Retention-bounded, rebuildable personal projection	Aggregate batch-training input or authentication proof
ANN artifact	Snapshot-owned representation, eligible index, mapping, checksum, optional query weights	An independently updated vector database
Training evidence	Versioned aggregate measurements about a run	Raw interaction history or business outcome attribution

There are now eleven trained strategies: the six original strategies plus five category strategies. For You is a serving strategy. The two recent-history strategies are Swimlane candidate sources. Geographic levels are variations of evidence, not four copies of each algorithm. The authoritative trained registry is STRATEGY_DEFINITIONS in domain.py; serving-only identifiers live there and in swimlane.py.

The distinction between eligibility, evidence, and authorization matters. Eligibility comes from the Catalog at the snapshot cutoff. Evidence determines candidate relationships and rank. Authorization governs access to property/Shopper context. Strong evidence does not make an ineligible Item returnable, and knowledge of a Product ID does not authorize Shopper state access. An Item made unavailable in the merchant source after publication remains governed by the old snapshot until a new generation is published; serving has no live inventory refresh path.

2. Decision map

The original choices and rejected alternatives are recorded in design decisions D-01 through D-17. That document describes the baseline, so read newer capabilities through current source and their dedicated contracts. The following table includes the cost, not just the benefit, of each choice.

Choice	Pressure it addresses	Cost or alternative deliberately left open	Basis
Separate API and worker	Batch CPU/I/O cannot own request latency or caller lifetime	Two processes and database coordination	Recorded D-01/D-02
PostgreSQL control store	Durable idempotency, claims, leases, publication transactions	Database becomes a shared availability and throughput dependency	Recorded D-03/D-11
Configured canonical SQL	Merchant schemas vary while algorithms need stable rows	Operators must certify query ordering, consistency, and source impact	Recorded D-04/D-05
Stream/reduce without raw staging	Preserve source ownership and privacy boundary	Reprocessing requires another source read; no service-owned raw replay archive	Recorded D-04/D-06
Whole-group exclusion	Pair work grows quadratically with context width	Discards useful pair evidence in oversized contexts	Recorded D-07
Regularized cosine and tier-first fallback	Normalize popularity and distinguish evidence strengths	More explicit policy; scores are not comparable across tiers	Recorded D-08
Sparse metadata baseline	Cold-item relationships without a remote model service	Depends on metadata quality and lexical/structured similarity	Recorded D-09
Immutable generation and atomic head	Keep a complete older answer during rebuild or failure	Storage amplification, staleness, cleanup, publication protocol	Recorded D-12/D-14
Bounded JSON set payload	Serve one ordered list with few indexed reads	Individual candidate analytics/updates are less relational	Recorded D-13
Polars as a second aggregate engine	Compare aggregate execution without rewriting source or serving	Duplicate engine implementations and a parity obligation	Inference; implemented by workstore_factory.py
Snapshot-owned ANN	Extend retrieval while keeping generation coherence	Native memory, cold loading, representation validation	Recorded ANN lifecycle contract
Export a NumPy query tower	Keep PyTorch training outside serving	Exported architecture/version must exactly match inference code	Inference from two_tower_torch.py and ann.py
Fail-safe telemetry seam	Exporter failures must not determine product outcomes	Monitoring loss can coexist with successful requests	Recorded operations contract

These are contextual decisions, not universal recommendations. A separate queue, distributed reduction, or external vector service could become appropriate if measured requirements exceed the current design. Such a change must preserve scope, privacy, and publication semantics first.

3. Source consistency is more than a timestamp

The worker fixes the evidence window and opens a source session through SourceAdapter.open. source.py supplies the canonical row types, streaming protocol, SQLAlchemy implementation, validation, and consistency token. Reviewed configuration supplies SQL; runtime scope/time values are bound parameters. API callers cannot submit arbitrary SQL.

A cutoff answers which event times are eligible. A consistent transaction answers which database state all queries observe. They are different questions: a backdated purchase inserted between the views and purchases queries could pass the same cutoff but create a mixed-state run. The source session and token exist to prevent that class of inconsistency. An engine being reachable through SQLAlchemy does not by itself certify its streaming, isolation, or ordering behavior.

Ordering is an algorithmic precondition. Catalog rows arrive in Product ID order; interactions arrive grouped by session or channel-scoped Order and ordered within the group. The reducer can then finish one group permanently when the next begins. Unordered input would require retaining many open groups or sorting raw records, both of which change the memory/privacy contract. Online and offline Orders with the same source ID remain distinct through PurchaseRow.group_id.

The entire Catalog is materialized for eligibility, metadata, anchors, and feature companions. Large interaction streams are reduced incrementally. Consequently, “streaming” does not mean constant process memory independent of Catalog size. A long source transaction also has a cost to the source system; acceptable plans, indexes, and transaction duration need deployment evidence.

Read source contract tests and raw-staging checks before modifying this boundary.

4. Why the training pipeline looks this way

One transient group, then identity-free aggregates

groups.py separates the view/purchase adapter from GroupAccumulator. The accumulator owns distinct membership, row count, latest timestamp, geography consistency, and the oversized flag. It drops membership immediately when the distinct Item bound is exceeded, but continues counting rows and checking group state. finish() emits a GroupSummary without the source Order/Session identifier.

ingestion.py owns derived buffering. Popularity counts each view and exact purchase quantity; pair evidence counts distinct membership once per context. Repeated visits to one Item can increase Most Viewed without strengthening a pair multiple times inside the same session. A group of m distinct Items implies m(m-1)/2 undirected pairs; this is why the default 100-view/200-purchase membership bounds are semantically consequential.

Oversized groups contribute no pair/support/category-pair evidence but retain popularity contributions. Keeping the first 100 Items instead would make evidence depend on stream order. Tests in group accumulation and group limits protect these semantics.

Two different backend selections

aggregation_backend chooses the complete work store. DuckDB is the default; Polars performs derived aggregation/scoring through its own temporary Parquet path. cooccurrence_backend chooses only the counting implementation inside the DuckDB work-store path. It can select native counting experiments without replacing source validation, popularity policy, or serving.

create_work_store rejects a nondefault co-occurrence override combined with Polars. This makes the distinction explicit rather than silently ignoring a requested backend. Both engines share the group reducer, so they must agree on quantities, exclusions, temporal partitioning, category constraints, geography, and final ranks. See Polars parity tests and end-to-end tests.

The default native writer can overlap derived work with serial source ingestion using bounded work submission. Parallelizing the source stream itself would require a new consistent-partition and group-ownership contract. Faster pair multiplication cannot fix a slow source cursor, expensive metadata construction, or publication bottleneck; measure phases separately.

Scores, selection, and fallbacks solve different problems

For pair support p and per-Item supports a and b, regularized cosine is p / sqrt(a*b) * p / (p + shrinkage). The first term discounts globally common Items; the second shrinks weak relationships. A minimum support threshold rejects weak pairs before ranking. These scores describe aggregate association, not purchase probability or causal influence.

Metadata Similar Items builds sparse category, brand, price, and text features, normalizes them, and uses bounded sparse cosine products. Popularity uses half-life decay; Trending measures recent momentum against a longer baseline. The formulas and parameter selection are in cooccurrence.py, content.py, and popularity.py.

The fallback chain traverses evidence tiers before comparing scores within a tier. A popularity score of 100 does not outrank a co-purchase score of 0.7 merely because 100 is numerically larger. Each emitted Item retains provenance. Deduplication, anchor exclusion, eligibility filtering, and stable Product ID tie-breaking happen at this boundary. See fallbacks.py.

Category-constrained strategies apply known-category equality/inequality before bounded behavioral top-k selection. Filtering a previously truncated global list could miss the best qualifying neighbors. Missing categories are not considered a shared category. These strategies do not widen to unrelated fallback evidence merely to fill a list.

Holdout evaluation and final publication use different evidence

The orchestration in run.py fixes a 28-day holdout and selects parameters from explicit grids. Whole behavioral contexts are assigned using their latest event time so an Order or session cannot straddle behavioral train/holdout membership. Popularity has its daily temporal aggregates. Selected parameters are then applied to full-cutoff evidence for the published generation.

Offline Recall/NDCG and coverage are proxy measurements. They do not establish business uplift. Catalog metadata is loaded at the run's source snapshot; a chronological interaction split alone does not establish historically versioned metadata at each past interaction. ANN representation quality and the learned retrieval objective need separate evaluation; existing batch metrics must not be relabeled as proof of those capabilities.

5. Publication is the consistency boundary

sequenceDiagram
    participant W as Worker
    participant R as Snapshot repository
    participant DB as Control database
    participant S as Serving
    W->>R: sets, features, optional ANN, manifest, run ownership
    R->>DB: create invisible building snapshot
    loop bounded batches
        R->>DB: stage sets and candidate features
    end
    S->>DB: read previous available head
    R->>DB: lock and check run/snapshot ownership
    Note over R,DB: One activation transaction
    R->>DB: insert ANN, mark available, replace head, mark run succeeded
    S->>DB: read newly published generation

The building state allows chunked writes without exposing a partial Catalog. The final transaction activates the Snapshot and records successful run state together. An exception during staging or activation discards building data and leaves the prior head available. Crashes can leave abandoned building state for recovery; an exception cleanup handler is not proof that code runs after a hard process termination.

In storage.py, follow publish, _stage_recommendation_sets, _stage_candidate_features, _lock_publication_run, _activate_snapshot, and _complete_publication_run. The worker passes both the active-run requirement and expected owner. A heartbeat alone is insufficient: a paused worker could resume after another worker reclaimed the run. Publication must check ownership at the durable boundary.

The manifest is compared with the entire RecommendationStrategy enum, currently eleven values. The worker publication path additionally checks observed strategy presence, requiring anchored strategies when eligible Items exist. This is not a repository-level proof that every possible anchor has every required row; generation and its tests also carry that obligation. Some legacy exception strings still say “six-strategy”; the enum comparison, not that wording, controls behavior.

Candidate feature companions now publish from the same Catalog read, including eligibility, category, and brand for eligible and ineligible Items. They are staged before activation and removed with failed/retained snapshots. This closes the former feature-population gap. It does not imply the Shopper projection already computes category/brand affinities; those are separate inputs to the reranker. See candidate storage tests.

The optional ANN artifact is validated before building the Snapshot and inserted in the activation transaction. A large ANN payload therefore still contributes to that transaction's size and latency; the transaction is not always just a tiny pointer update. Immutable does not mean retained forever: retention preserves the current head and removes older complete generations.

6. Serving is source-free, not computation-free

Path	Reads	Computes at request time	Fallback boundary
Ordinary strategy GET	Published head and bounded set	Validation, slice, stale metadata	Explicit missing/empty/incomplete states
Personalized lane	Published candidates/features plus authorized profile	Bounded rerank and concentration limits	Ordinary lane
For You without ANN	Bounded global union plus profile/features	Personalized rerank	Best Sellers
ANN Similar Items / For You	Selected generation's validated artifact and query inputs	Query representation and eligible vector search	Existing snapshot path
Contextual Swimlane POST	One head's variations, optionally authorized typed recents	Ordered accumulation, recent-seed fusion, history fill	Later configured steps, then history

serving.py contains application decisions and typed failures. SnapshotLaneLoader and ForYouCandidateLoader distinguish candidate sources from ordinary fallbacks. AuthorizedPersonalization and UnavailablePersonalization make unavailable runtime behavior explicit. api.py owns transport, authentication plumbing, status mapping, and response projection. This is a useful seam because HTTP errors should not be the internal representation of ranking decisions.

An empty, valid set means insufficient evidence. A missing head means no successful generation. A missing required global lane or invalid payload means incomplete/corrupt serving state. Those states should not collapse to an empty success response. Staleness uses generation time plus the freshness policy and does not trigger synchronous training.

Snapshot consistency is per operation, not an implicit distributed transaction over every helper. The contextual resolver explicitly uses get_head_variations for a single-head read. The current For You loader fetches its global union and Best Sellers fallback through separate repository calls; do not assume these calls share a transaction during concurrent publication. This is a remaining consistency concern to test, not evidence that a mixed response was observed here.

7. Geography, recent history, and Swimlanes

GeoKey normalizes source-local names using NFKC, case folding, and whitespace collapse; city requires region and country. This differs deliberately from NFC identity normalization for opaque scope/Product identifiers. Normalization reconciles encodings, not geographic aliases. A same-name city in a different parent remains distinct. No network geocoder is involved.

Groups contribute to their available hierarchy and global aggregates. Missing geography contributes globally only; conflicting geography inside a group is rejected. Local publication requires ten qualifying contexts, with behavioral minimum pair support at least ten. Local sets do not borrow metadata/global candidates. Sparse local absence means the configured chain may continue; it does not mean the global strategy failed. The ten-context threshold is not a ten-person anonymity claim.

The current local recipes produce co-view/co-purchase and category-constrained anchored sets. Configuration can express other strategy/level combinations, but that does not manufacture an unpublished geographic variation. In particular, do not assume a local Similar Items, Trending, or global-popularity strategy set exists merely because those identifiers parse.

resolve_swimlane in swimlane_service.py compiles each step's required lookup keys, reads them under one published head, and passes lazy candidate providers to compose_swimlane. The accumulator owns quota, deduplication, anchor exclusion, and step-specific purchase exclusion. This separates selection policy from storage and avoids repeatedly deciding geography/seed meaning during consumption.

For quota four, if the city step supplies [A, B] and the region step [B, C, D], the result is [A, B, C, D]; B consumes no second slot. A higher-scored regional Item cannot jump ahead of A. History is consulted only after all configured steps leave room. It uses newest-first authorized recent views and snapshot feature eligibility. Missing features do not authorize a history Item.

Recent-view and recent-purchase seeds remain distinct. The resolver uses at most ten seeds, loads published Also Viewed/FBT neighborhoods, keeps only matching co-view/co-purchase evidence, and fuses scores with reciprocal seed-position weights. A popularity fallback inside FBT is therefore not silently reinterpreted as a co-purchase relationship. Purchase exclusion is a configured step filter, not a universal ban on replenishment.

The contextual endpoint authenticates a property bearer before parsing the body. Shopper context is needed only for profile access. Location selects aggregate evidence; it is not identity proof. Responses omit local exact scores/support and use private, no-store. The endpoint is POST because context belongs in a body rather than routinely logged URLs; it is still a read-only operation. See Swimlane tests, resolver tests, and geographic publication tests.

8. Embedded ANN and the two-tower option

Retrieval, representation, and relevance are separate

ANN answers which vectors are near a query efficiently. Representation decides what proximity means. Neither alone establishes recommendation quality. The metadata baseline builds deterministic truncated-SVD projections from the existing sparse feature matrix, normalizes finite float32 vectors, and constructs a CPU Faiss HNSW index over eligible Items. The representation may retain vectors for other Items as query inputs, but return eligibility comes from the indexed generation.

The implementation in ann.py uses HNSW connectivity 32, construction search 160, and default query search 512. These are implementation settings, not proven optimal values. Small eligible indexes (at most 2,000 Items) use exact search; ANN underfill also falls back to exact search. This improves quota recovery after exclusions at the cost of a possible Catalog-wide vector scan. Exact search is chunked, but allocates a score array proportional to eligible Catalog size.

Artifact validation checks representation version, shape, finite values, mapping/index consistency, and checksum. The SHA3-512 digest detects mismatched/corrupt content; it is not a signature proving who produced it. Artifact ownership, access control, and snapshot scope remain separate concerns.

What the learned option actually learns

two_tower_torch.py trains two independent two-layer MLPs on metadata projections. Positive pairs come from already bounded aggregate co-purchase and co-view candidates; purchase candidates receive twice the selection weight. The implementation selects at most 10,000 pairs, caps dimensions at 128, runs three epochs in batches of 128, and uses in-batch contrastive cross-entropy with AdamW. It chooses MPS when available and CPU otherwise.

This is aggregate Item-relationship training, not a model trained on raw Shopper sequences. The pair selection score chooses examples; the training loss does not thereby become a continuously weighted purchase-value objective. Fixed seeds make controlled repetition easier, but do not prove bitwise equivalence across native libraries/devices. Co-purchases can represent complements, so learned proximity needs separate scrutiny before being presented as substitution quality.

The worker exports Item vectors, original projected input vectors, and query-tower parameters. Serving computes the small query network using NumPy, then searches the immutable Faiss index. PyTorch is an optional training dependency. Faiss is a base package dependency but is imported lazily on opted-in build/load paths; “default off” is not the same as “absent from the wheel's dependency graph.” Inspect pyproject.toml for the actual packaging boundary.

Why the index belongs to the Snapshot

Independent online index updates could mix eligibility, representation, and Item-ID mappings from different Catalog generations. Snapshot ownership makes one generation the retrieval authority. Acknowledged Shopper interactions can change the query between Training Runs; they do not mutate the index. For You excludes recent/negative Items from ANN candidates and can discover beyond the old bounded global union. Metadata and learned results have distinct provenance and low confidence; distance is not a calibrated probability.

The API assembles AnnSnapshotRetriever with background loading. It retains at most two loaded generation entries by default and admits at most one pending load. A cold/missing/invalid artifact returns the ordinary path while warming. The cache is shared within that retriever, not two generations per Commerce Scope, and eviction follows insertion order rather than access-based LRU. Failures are remembered for that generation until eviction or restart. Those details matter for a multi-scope deployment's hit rate and recovery behavior.

Inspect ANN snapshot tests, two-tower tests, and ANN experiments. Index recall, representation preservation, personalized relevance, full-request latency, and native memory residency require different evidence.

9. Personalization has its own consistency and privacy lifecycle

The batch worker reduces merchant interaction streams without retaining raw rows. The optional personalization API instead accepts purpose-authorized, bounded interactions into a retention-bound ledger. These are distinct ingress contracts; the ledger is not permission to copy merchant source history into the control database.

An interaction transaction commits its ordered ledger entry, idempotency record, lifecycle barrier, and profile projection before an acknowledgement is issued. Exact retries replay the acknowledgement; different content under the same interaction identity or a wrong next sequence conflicts. A causal token requires observation of at least the acknowledged projection version within a bounded wait. It is neither authentication nor a promise to wait indefinitely.

Opt-out/deletion barriers prevent use immediately while bounded cleanup removes retained state. Rollback of read/write admission must not disable deletion and expiry reconciliation. The worker calls personalization maintenance even when it finds no Training Run; an idle training queue is not permission to stop the worker. Reusing the worker avoids a separate cleanup process, but ties maintenance cadence to worker availability and long-running work. That operational consequence is an inference from worker.py, not a measured deletion SLO.

Snapshot category/brand features enable concentration controls. The current interaction projector does not populate category/brand affinity maps from Item interactions, so feature publication alone does not complete every possible personalization signal. The full contract and release gates are in Shopper personalization.

10. Resource and operational costs

Resource	Current control	What the control does not establish
Raw context membership	Configured distinct-Item bounds; clear on oversize	Total Catalog/derived-pair memory
Derived buffering	Batch-key limits and bounded work submission	A process-wide RSS ceiling
DuckDB scratch	Engine memory/spill settings, per-run cleanup	Identical behavior in the Polars engine
Polars execution	Derived-file budget and engine-specific checks; native thread pool	DuckDB-style spill guarantees or a resize from pipeline.threads
Result payload	At most 100 entries per set	Constant total Snapshot size
Geographic output	Current 50,000 local-set cap	Complete publication of every supported local partition
Swimlane	At most 32 steps, quota 1–100, bounded seeds/history	Every configured step having available evidence
ANN residency	Two cached entries and one pending load by default; payload budget	Total native/Python peak RSS or fleet-wide residency
Admission	Process-local token buckets/concurrency leases when injected	Distributed capacity enforcement

With E eligible Items, the global registry emits eight anchored sets per Item and three global sets, including explicit empty outcomes: roughly 8E + 3 set rows before geographic variations. Each row can hold up to 100 entries. Retaining multiple generations multiplies storage again; candidate features and optional ANN add per-generation costs. Bounded per-row work is not a claim that a full run has constant memory or trivial storage.

The pipeline materializes Catalog features, candidate maps, and resulting sets in addition to work-store scratch. For vectors alone, N * d * 4 bytes describes one float32 matrix, before the native index, mappings, temporary copies, and (for two towers) exported input vectors. Concurrent old/new generations and loading can increase peak memory beyond a single artifact's byte budget.

Operationally, separate these questions: Is the source slow? Is reduction CPU-bound? Is scratch exhausted? Is publication slow? Is serving waiting on the control database? Is an ANN generation warming? A single training-duration number cannot distinguish them. The typed observability seam and operations contract expose bounded phase/outcome evidence.

Quality drift compares same-scope successful runs with compatible model/config/evidence versions, using bounded history and robust baselines. Assessment is diagnostic after publication. A warning does not reject a completed generation or establish conversion degradation. Exporter failure also must not change product behavior, though it can remove the evidence operators need.

11. Why there are both generated sources and a simulation harness

The generated source provides deterministic relational contents, a logical identity, manifests, digests, and engine materialization receipts. It is suited to reproducible source-to-serving oracles and controlled scale experiments. The simulation source is mutable and run-isolated; a storefront and bounded visitor runner create activity over logical time and observe training checkpoints. One is not a replacement for the other.

Independent simulation behavior does not follow recommendations. Feedback behavior can follow them and therefore changes its own future training evidence. A successful feedback scenario proves that a loop operates; it does not prove independent relevance or causal uplift. A source receipt proves the materialization matches its manifest, not that the service meets production latency.

Read synthetic generation, generated-source operations, and simulation contracts. These guides also explain append lineage, atomic replacement, reproducible random streams, and checkpoint attribution in more detail.

12. Known limits and design drift

These observations bound the claims in this guide. They do not authorize changing runtime behavior or silently revising product intent.

Observation at the audited revision	Consequence and next verification boundary
_build_geographic_sets returns when 50,000 sets have accumulated	Later local results are omitted; the draft design's fail-on-budget-exhaustion proposal is not implemented. Evaluate coverage and explicit failure semantics before relying on full geographic coverage.
Geographic budgets are constants/implementation controls rather than the draft's full configurable partition/pair/publication policy	Do not advertise the draft's complete resource contract as shipped.
For You candidates and ordinary fallback use separate head reads	Concurrency testing should establish required behavior across a head swap; only contextual variation lookup explicitly pins the combined lookup.
Candidate features now publish, but profile category/brand affinities are not derived	Concentration metadata exists; affinity behavior is a separate implementation boundary.
Repository publication checks strategy presence, not a Cartesian proof over every anchor	Pipeline generation and integration coverage remain part of completeness assurance.
ANN opt-in has bounded artifacts and tests, not broad deployment qualification	Measure representation, relevance, memory, cold load, underfill, and request latency separately.
Ordinary training/GET APIs have no general built-in caller authentication	Deployment/network integration owns that boundary; contextual/personalization authorization is separate.
SQLite/local tests are part of the normal loop	They cannot establish PostgreSQL lock races, production availability, or business outcomes.

The geographic design remains a draft with explicit reconciliation notes. No new ADR is manufactured here: an explanatory reconstruction must not pretend that an inferred reason was an approved historical decision. Future decisions should record the actual alternatives, constraints, and evidence at the time they are made.