Testing and local verification

Feedback ladder

Run the narrowest command that can disprove a change, then widen before handoff.

Scope	Command	Use it for
Setup	make setup	Create/update the locked development environment
Diagnosis	make doctor	Check tools, Python, lock consistency, sibling telemetry, and dependencies
One test	make test-focused TEST=tests/path.py::test_name	Red-green work and failure reproduction
Style	make lint	Ruff correctness, imports, formatting drift, and documentation rules
Types	make type-check	Strict package contracts
Repository	make docs-check architecture-check	Links, command surface, and import boundaries
Documentation site	make docs-site	Build the Pages artifact and validate site links
Unit	make test-unit	Pure algorithms, schemas, policies, and bounded state
Contract	make test-contract	HTTP, source-adapter, privacy, and repository contracts
Integration	make test-integration	SQLite lifecycle, migrations, and end-to-end component behavior
PostgreSQL	make test-postgres	Locking, generated-source verification, and atomic publication on a disposable database
Installed wheel	make smoke	Build metadata, dependency installation, imports, and entry points outside source
Normal handoff	make test	Static, unit, contract, and non-PostgreSQL integration gates
Complete local	make verify	Normal handoff plus clean-wheel smoke test

make migration-check isolates migration round trips. make compose-check validates both the simulation stack and the generated-source Compose file using a check-only password; it does not start PostgreSQL or generate data. A full simulation run is described in commerce-simulation-harness.md.

Bootstrap contract

Python 3.14 is the authoritative development version in .python-version; the distribution supports Python 3.12 through 3.14. The uv.lock file resolves the telemetry dependency from the sibling path ../telemetry, so both repositories must share a parent directory:

workspace/
├── recommendations/
└── telemetry/

make setup is non-interactive and installs the locked development, PostgreSQL, observability, and PyTorch extras. Run make doctor after setup or when the environment behaves differently from CI.

For the reasoning behind test placement and a concrete contract-to-code change sequence, read implementation workflow. For documentation-only edits, run make docs-check and the relevant delivery contract tests; behavioral, database, and performance suites are needed when those boundaries change. The link validator checks file existence, not heading anchors, factual accuracy, or whether a Mermaid diagram renders.

Published documentation

make docs-site builds the documentation website in .build/docs-site/ using pinned MkDocs and Material versions. The builder stages selected root Markdown and the docs/, design/, and specs/ trees, including the standalone benchmark report. It does not publish service source or configuration files.

Links to source files excluded from the site open the corresponding file on GitHub. Mermaid blocks render as diagrams. The strict build checks site navigation and local links.

The documentation site workflow builds pull requests and deploys documentation changes from main to Cloudflare Pages; it can also be run manually from main. Create a Cloudflare Pages Direct Upload project named recommendations-docs, then configure the repository's Actions secrets CLOUDFLARE_API_TOKEN (an account token with Cloudflare Pages Edit permission) and CLOUDFLARE_ACCOUNT_ID. Only the generated site is uploaded; Cloudflare does not need access to the private repository. The workflow uses a GitHub-hosted runner and does not require the sibling telemetry checkout. The public site is https://recommendations-docs.pages.dev/.

Test layers and state

• tests/unit/ exercises algorithms and policies without live services.
• tests/contract/ exercises public HTTP, source, privacy, and observability boundaries.
• tests/integration/ exercises lifecycle and persistence across components. PostgreSQL-marked cases skip unless TEST_CONTROL_DATABASE_URL is explicitly configured.
• tests/delivery/ protects the repository harness, CI shape, and agent-facing artifacts.

Tests construct state per case. Synthetic generators and simulation scenarios use explicit seeds and logical time. Use only disposable databases for destructive migration or PostgreSQL tests. The PostgreSQL synthetic-source integration cases exercise concurrent COPY, a single generation pass, byte-identical manifests, optional empty tables, matching CLI artifacts, and preservation of published tables after a failed replacement. Set TEST_CONTROL_DATABASE_URL to an explicitly disposable PostgreSQL database and run make test-postgres; the fixture drops generated-source tables after each case. make test excludes these cases.

Append coverage checks preservation of historical rows and Catalog metadata, interval rates and timestamps, disjoint context/row identities, repeated cutoffs, export recovery, wrong-scope and concurrent-writer rejection, full rollback on COPY or verification failure, and lineage reset after replacement. These cases use only the disposable PostgreSQL test database; normal verification does not regenerate or append to the developer's generated source.

make test-focused TEST=tests/unit/test_workstore.py verifies bulk derived insertion with synchronous and background writers, exact decimal quantities and dates, whole-group holdouts, oversized-group exclusions, source/write overlap, backpressure, error propagation, and scratch cleanup. It also compares duckdb, sparse, and accelerate pair/support strategies with the independent Python reference reducer, verifies concurrent native batches and identity-free worker inputs, and exercises multiple buffer sizes and thread budgets. tests/integration/test_end_to_end.py compares complete serial/parallel recommendation results across all three backends and preserves the serving head after a native-task failure. tests/unit/test_sparse_cooccurrence.py verifies strategy selection, binary membership semantics, multiplicity weighting, empty inputs, and unsigned 64-bit counts/overflow. The Accelerate cases compare large integer counts with the SciPy backend on macOS. Configuration tests cover the default, explicit selection, invalid values, and configuration-fingerprint changes. The optional torch, mlx, and ane backends each have focused exact-count tests; those cases skip when the extra is absent or the host is not macOS. Install all extras to exercise them locally: uv sync --extra dev --extra torch --extra mlx --extra ane.

Training performance checks

make test-focused TEST=tests/unit/test_polars_workstore.py compares the full Polars aggregate engine with DuckDB across batch sizes, temporal partitions, oversized groups, online/offline purchases, exact decimal quantities, geography, category filtering, and ranking. It also checks empty inputs, repeated ingestion/queries, scratch cleanup, configuration fingerprints, resource failures, and that Polars aggregation opens no DuckDB connection. The Polars cases in tests/integration/test_end_to_end.py compare complete recommendations/evaluation and verify worker selection, failure classification, and preservation of the published serving head. These are correctness checks; Polars memory/spill behavior and performance still need independent qualification on the target deployment.

make benchmark-ann ANN_BENCHMARK_ARGS='--items 2000 --queries 40' runs a bounded, synthetic Faiss Flat versus HNSW comparison with native Item-ID filtering. The target adds a pinned Faiss CPU wheel only to that experiment process; it does not change the application lockfile. Defaults are a small smoke workload. Set --items 200000 explicitly for the Catalog-size experiment and record available RAM and thread settings. The JSON report distinguishes exact-search and HNSW timings, recall against exact search on the same synthetic vectors, eligible quota fill, build time, serialized index size and process peak RSS. It does not measure the proposed metadata representation, recommendation quality, full Serving API latency or deployment qualification. Native compile options are reported as evidence of the installed wheel; they do not prove that a particular query used the Apple GPU or Neural Engine.

make benchmark-ann-representation separately compares deterministic dense SVD projections of the existing metadata features with the current exact sparse Similar Items results on the committed smoke Catalog. The default dimensions are 64, 128, 256 and 512; use ANN_REPRESENTATION_ARGS='--components 128 256' to narrow them. This is a bounded representation experiment, not ANN recall or merchant-Catalog quality.

The co-occurrence benchmark suite runs every selected counting backend in fresh processes across varied group widths, catalog sizes, overlap patterns, thread budgets, and an explicit larger-than-memory-budget profile. It verifies exact aggregate signatures and reports elapsed time, CPU, peak RSS, page faults, disk footprint/spill, and optional system-level power estimates. Use make benchmark-cooccurrence-suite SUITE_ARGS='--profile smoke' for a safe functional check before running the larger profiles.

make benchmark-training BENCHMARK_ARGS='--threads 4' reduces one million deterministic synthetic views (50,000 groups of 20), verifies aggregate totals, and prints elapsed time, process CPU time, average CPU cores, and throughput. It opens no external source, retains no raw rows, and cleans its derived scratch on exit. CPU time includes all threads in the process; one CPU-second per elapsed second corresponds to one fully occupied core. Verification queries are outside the timed interval. Use BENCHMARK_ARGS='--aggregation-backend polars --threads 4' to compare the complete Polars aggregate path with DuckDB on the same generated views. Set POLARS_MAX_THREADS=4 in the environment before the Polars process starts; --threads alone cannot resize its native pool. Run each backend in a fresh process, with identical --groups, --group-size, and --batch-key-limit. This is a reduction benchmark, not a full Training Run qualification.

Compare --threads 1, --threads 4, and --threads 8 sequentially on the same machine. Use --groups 100000 --group-size 3 for small groups or --groups 10000 --group-size 100 for pair-heavy groups. --batch-key-limit controls the same bound as PipelineConfig. These are local reduction microbenchmarks, not end-to-end or capacity qualification: source validation/I/O, catalog similarity, parameter selection, and publication are not included. See the implementation plan and evidence.

Add --cooccurrence-backend duckdb (the default), --cooccurrence-backend sparse, or (on native macOS) --cooccurrence-backend accelerate, torch, mlx, or ane to compare counting implementations with identical inputs and bounds. The JSON report records the selection. Compare elapsed time as well as CPU usage: more occupied cores do not necessarily mean faster training, and source streaming/validation remains serial for all implementations. GPU/ANE backends may lower measured CPU use while increasing wall time on small batches due to dispatch and model compilation overhead. ANEForge is experimental and relies on private Apple interfaces.

On an M2 Pro with four configured threads, a local single-run comparison verified identical aggregate totals for all three backends. For 50,000 groups of 20, elapsed times were 3.92 s (duckdb), 4.21 s (sparse), and 7.23 s (accelerate). For 1,000 groups of 100, they were 0.54 s, 0.54 s, and 1.01 s respectively. These measurements do not establish a general ranking; they show that the current Accelerate path is an opt-in experiment rather than a speed improvement on either measured workload.

Regression discipline

make setup installs the optional PyTorch extra for the opt-in two-tower training tests. The published base package does not require PyTorch: default-off Training Runs and API serving import only the exported NumPy/Faiss artifact. Apple MPS execution and performance still require a separate qualification run on the target machine.

For a meaningful defect, first identify a focused command that fails on the defective behavior. Add a regression test that demonstrates that failure, apply the fix, and rerun the same command before widening the suite. Record both states in the handoff. A new passing test without red-state evidence is weaker and needs an existing contract or explicit review of the failure mechanism.

A flaky test requires a tracked issue with its observed failure rate and suspected layer. Any temporary skip links that issue. Infrastructure failures and product defects receive separate diagnoses even when they fail the same CI job.

Generated artifacts

The repository has no generated source. .build/, artifacts/, dist/, coverage files, caches, virtual environments, generated simulation reports, and generated relational sources are outputs. Regenerate packages with make build, the installed-package report with make smoke, OpenAPI with scripts/export_openapi.py, and simulation evidence through the documented harness. Do not edit generated output as source.

CI on the Mac mini

The workflow selects [self-hosted, macOS, ARM64], verifies the runner architecture, and pins Python, uv, action revisions, and the telemetry commit. The runner needs:

• all three labels, Git, and Xcode command-line tools;
• network access to Python package indexes and GitHub Actions;
• repository read access through the TELEMETRY_REPOSITORY_TOKEN Actions secret; and
• enough free disk for uv environments and compiled scientific wheels.

The workflow checks both repositories into sibling directories in every job so the lockfile's local telemetry source resolves identically to development. It serializes quality, package, and installed- package compatibility evidence for the single runner. The compatibility matrix exercises the same wheel on Python 3.12, 3.13, and 3.14. Jobs retain JUnit, branch coverage, package digests, runner identity, and the clean-wheel smoke report for 14 days. The workflow has read-only repository permissions and no publishing or deployment authority. Pull requests from forks are skipped because GitHub does not expose the private telemetry checkout token to them.

Required-check selection and branch protection are hosted settings. Configure the stable job names Quality and tests / Python 3.14, Package contracts / Python 3.14, and the supported Installed package / Python <version> matrix checks after the workflow succeeds once.