Implementation process and change recipes

This document explains how to work on this implementation and why the process follows its boundaries. It is a practical companion to engineering rationale, testing, and the repository's change contract.

The workflow below is the supported way to make a change. It is not a claim that every historical commit followed the same sequence. Recorded design plans, commit subjects, and executable tests provide different kinds of evidence; none alone reconstructs an author's entire process.

1. Establish the behavior before selecting the mechanism

Begin with a concrete observation: a request and expected result, a failed-run recovery case, a source row that must be rejected, or a resource bound that must hold. Read CONTEXT.md, then the governing spec and existing tests. Determine whether the request changes intended behavior, repairs implementation drift, or only changes the mechanism behind an unchanged contract.

For example, “make training use Polars” is not permission to change pair counts, quantity precision, holdout membership, or rank ordering. Those remain observable contracts. “Add same-category recommendations” does change intent: the implementation must decide what missing categories mean, when filtering happens, and whether fallback can leave the category before choosing an algorithm.

Inspect git status --short before editing. Preserve unrelated changes. Use docs/index.md to find the right authority; a research document or a draft design is not a substitute for approved behavior. Current code/migrations/tests establish what exists; specifications establish what is intended. Report a mismatch explicitly instead of concealing it by rewriting one as though the other had always agreed.

2. Trace one operation end to end

Operation	Entry and application path	Durable or external boundary	Evidence to inspect
Submit training	api.py → training repository	Scope allowlist, idempotency, active-run uniqueness	API contracts, lifecycle
Execute training	worker.py → pipeline/run.py	Consistent source session, work store, lease	source adapter, end-to-end
Publish generation	SnapshotRepository.publish	Invisible stages and atomic head/run activation	PostgreSQL storage, candidate companions
Ordinary/personalized serving	api.py → serving.py	Published head, bounded profile/features	serving unit tests, API contracts
Resolve Swimlane	api.py → swimlane_service.py → swimlane.py	One-head variations and authorized history	composition, resolver
ANN retrieval	Pipeline builder → artifact → AnnSnapshotRetriever	Checksummed generation, native load/cache	ANN lifecycle, two towers

Follow ownership as well as calls. Identify who fixes the cutoff, who can advance the serving head, who authorizes profile reads, who cleans scratch, and who translates failures. A helper that returns the expected value can still be wrong if it reads from the wrong generation or bypasses a barrier.

For every change, enumerate the relevant failure path: source failure after partial ingestion, lease loss after computation, failure after a feature batch, missing ANN payload, expired causal token, or concurrent head advancement. This often reveals the correct seam sooner than a class diagram alone.

3. Place policy where its inputs and invariants belong

Use pure functions or domain values for deterministic policy; use an object when it owns state, lifecycle, or a useful variation boundary. Existing examples are deliberate:

• GroupAccumulator owns one transient context; view/purchase adapters supply different grouping and weighting semantics.
• compose_swimlane owns quota and ordered selection; its providers own authorized snapshot reads.
• create_work_store selects a complete engine once; backend branches do not belong in every source-row consumer or API handler.
• RecommendationServingService returns typed application results; HTTP status codes and telemetry projection remain in the API adapter.
• AnnSnapshotRetriever encapsulates load/residency/fallback behavior; callers need not manage native index internals.

These are implementation inferences about why the current seams are useful. They are not a mandate to add a protocol around every function. Share a module when it owns a stable rule; do not share merely because two loops have similar syntax. Preserve serving_limits package independence and the recommendations.observability telemetry adapter boundary.

4. Build evidence at the lowest meaningful layer

Choose an assertion that could fail if the intended behavior were broken. Examples:

Risk	Useful observable test	Insufficient substitute
Oversized context partially leaks pairs	Assert whole-group exclusion while popularity remains	Assert the configured limit constant
Category filtering occurs too late	Best matching candidates lie outside unfiltered top-k and still appear	Filter a list where every candidate already matches
Publication failure exposes new state	Stage data, inject failure, assert old head and cleanup	Assert publish was called
Geography crosses parent boundaries	Same-name cities under different parents stay distinct	Use only one city
Recent views seed purchase strategy	View-only history supplies no bought seeds	Assert both fields exist on a profile
ANN cold loading blocks a request	Hold the loader and observe immediate ordinary fallback	Time only a warm native index query
A new engine changes semantics	Compare independently known quantities/ranks and cross-engine results	Compare two wrappers calling the same algorithm

For a code defect, capture a failing regression where practical, then apply the fix and rerun that test. Backend parity alone is not an independent correctness oracle: two engines can implement the same mistaken interpretation. Keep explicit examples with known counts and ranks alongside parity.

Use fixed clocks, seeds, logical time, and disposable per-test state. Avoid production identifiers and credentials. Tests crossing a persistence/HTTP/native boundary belong at that boundary; do not replace its behavior with mocks and then claim the integration is proven.

5. Recipe: add a trained strategy

1. Specify the shape, primary signal, fallback order, eligibility, missing-data behavior, and evidence/provenance meaning. A new strategy is not automatically a new backend.
2. Add the trained identifier and StrategyDefinition in domain.py. Decide whether geography has an actual recipe; parsing a strategy/level is not enough to produce local sets.
3. Build candidate evidence in pipeline/run.py and the relevant pure algorithm/work-store layer. Apply semantic filters before bounded top-k where required.
4. Emit all required global/anchored outcomes, including explicit empty sets. Extend holdout truth, evaluation, metrics, and baseline comparisons for the intended relationship.
5. Verify both work-store engines if aggregate queries change. Preserve source ordering, exact quantities, and identity-free derived state.
6. Check manifest/publication behavior and old-head serving during rollout. The current complete manifest follows the enum, so adding a value changes what a newly published run must provide.
7. Extend API/contract tests, operations vocabulary where relevant, and current documentation.

A serving-only recent-history source instead belongs to RecentStrategy, step compilation, and composition. For You belongs to the serving registry. Do not add fictitious batch output to satisfy a manifest for behavior that is intentionally computed at consumption time.

6. Recipe: evolve a snapshot companion

Candidate feature publication is a useful concrete example already present in the code. The pipeline derives Product ID, category, brand, and eligibility from the same Catalog read as the sets. PipelineResult carries these values. The worker passes them to publish, which stages them in bounded batches before activation. Serving retrieves them by the selected Snapshot ID.

That sequence protects against a tempting shortcut: fetching live category/brand metadata during serving would avoid the companion table but break generation consistency and source isolation. Populating features after head activation would expose a temporarily incomplete generation.

When adding another companion, define its version, scope, completeness rules, size limits, absent payload behavior, and cleanup. Update runtime SQLAlchemy metadata and a new Alembic migration when schema changes. Verify failed staging, activation rollback, generation/scope isolation, retention, and older snapshots without the companion. Do not silently rewrite an applied migration.

7. Recipe: change performance without changing results

Start with a bounded workload and an identified expensive phase. Separate source I/O, validation, group reduction, scoring, metadata construction, ANN work, and publication. Record the input shape: Catalog size, context width, pair overlap, backend, thread budgets, and scratch constraints.

First prove exact counts and externally visible rank/provenance parity. Then compare wall time, CPU time, peak RSS, and scratch usage in fresh processes. Native libraries can retain thread pools, caches, or compiled state across runs. More CPU utilization is not proof of lower elapsed time; GPU dispatch can cost more than small-batch computation saves.

Use the committed make benchmark-training, make benchmark-cooccurrence-suite, make benchmark-ann, and make benchmark-ann-representation targets with deliberately bounded arguments described in testing. Do not run qualification-sized experiments merely to verify a small change. Record representation quality separately from ANN index recall, and both separately from full-request latency. Benchmark success does not change production rollout authority.

8. Verify in proportion to the changed boundary

The supported command surface is Makefile. Start focused, then widen through the relevant gates; the table is a selection guide, not an instruction to rerun the same suites repeatedly.

Change	Initial evidence	Wider applicable gate
Documentation only	Source/spec comparison, git diff --check	make docs-check, relevant delivery tests
Pure policy/domain	make test-focused TEST=...	make test-static, normal make test
HTTP/source contracts	Focused contract tests	make test-contract, make test
Persistent schema	Migration/lifecycle tests	make migration-check, make test; disposable PostgreSQL checks when relevant
Publication/leases	Failure injection and concurrency contracts	make test-postgres with explicit disposable URL
Packaging/dependencies	Installed artifact checks	make smoke / make verify
Docker/Compose	Configuration validation	make compose-check
Performance claim	Correctness plus bounded measurements	Approved workload/environment qualification

make setup updates the locked environment; it is not necessary for every prose edit. make doctor diagnoses prerequisites. Python environment selection must match the configured project interpreter. PostgreSQL tests must never guess a disposable database from an unrelated development connection.

Documentation checks validate local target files and the Make command surface. They do not prove that a heading exists, a diagram renders, an architectural assertion is true, or a recipe is safe. Review those manually against source and tests. Avoid fragile source line anchors in explanatory guides; name the symbol and link the module.

9. Keep explanation, specification, and history honest

Update documents at the boundary that changed. Use CONTEXT.md for concise vocabulary, specs/ for observable intent, design/ for decisions and alternatives, the technical reference for current mechanics, and operations docs for deployment/diagnosis. Explain why a choice exists and what it costs; do not merely restate function names in prose.

Recorded reasons can be linked to their decision. Inferred reasons should say so. A commit subject such as “implemented ANN + two-towers” establishes that work was introduced; it does not prove the model was quality-qualified or that all proposed release gates passed. A historical baseline can retain its original six-strategy design while a prominent note points to the current eleven- strategy implementation. This preserves history without misleading a new reader.

For this documentation audit, the process was: check the clean working tree; read vocabulary and governing docs; trace current registry, worker, publication, serving, reducer, and ANN code; inspect the associated tests; reconcile stale claims; add rationale and current limitations; then validate links and the relevant delivery harness. Runtime source and behavior were left unchanged.

10. Hand off reviewable evidence

A useful handoff states the changed behavior or explanation, names the contract evidence, lists commands and results, and names checks not run with reasons. Keep “source inspected,” “test exists,” “test passed here,” and “qualified in a deployment” as different claims. Use docs/agents/handoff.md when work crosses a session or agent boundary.

Review the final diff for unrelated edits, accidental secrets, stale counts/version claims, links to moved symbols, and diagrams that overpromise consistency. If a known implementation limitation remains, state its precise consequence and the boundary that would verify a future change. Do not turn a documentation pass into an unrequested runtime redesign or imply that a recorded limitation has been fixed.