Commerce recommendations¶

An internal, multi-property item-to-item recommendation service. It streams catalog, view, online-purchase, and optionally in-store/offline-purchase rows directly from configured relational databases, trains eleven batch strategies, atomically publishes immutable snapshots, and serves pre-ranked results. A default-off additive capability can rerank bounded candidates for an authorized Shopper without changing the underlying snapshot.

Implemented strategies:

Frequently Bought Together: Order-level co-purchase with regularized cosine, followed by complementary category and purchase-popularity fallbacks. Metadata substitutes are excluded.
Also Viewed: Session-level co-view with regularized cosine, followed by metadata similarity, category views, and property views.
Similar Items: sparse metadata TF-IDF/cosine.
Best Sellers, Most Viewed, and Trending: time-decayed global rankings.
Five category-constrained anchored strategies: popular viewed/bought in the same category, frequent bought in a different category, and frequent viewed in the same/different category.

Configured Swimlanes accumulate strategy/geographic variations in order, then use authorized recent view history to fill remaining slots. Separately enabled personalization can rerank candidates; opt-in embedded ANN can retrieve snapshot-local Similar Items and For You candidates using metadata SVD or an aggregate-trained two-tower representation. These paths still never query merchant data while serving.

The approved product specification is specs/commerce-recommendation-service.md; the implementation design is design/commerce-recommendation-service.md. The implementation-as-built reference, including UML sequence diagrams, class diagrams, durable and ephemeral ERDs, algorithms, lifecycle, failure handling, and operations, is docs/technical-implementation.md. Use docs/index.md to navigate the complete documentation set and docs/command-reference.md for the commands needed to build, test, simulate, generate data, benchmark, and deploy documentation. Use AGENTS.md for the repository change and verification contract. For a start-to-finish explanation of the repository, including code excerpts and the rationale behind its boundaries and integrations, read the Commerce Recommendations repository guide. For an experienced engineer's explanation of the current choices and their costs, start with engineering rationale, then the implementation workflow. These distinguish recorded decisions, code-derived explanations, and implementation limitations.

A reusable request-rate and concurrency admission-control foundation is described in design/serving-admission-control.md. It is optional and disabled in the recommendation API unless a policy controller and trusted context factory are provided; its in-memory adapter is not a distributed-capacity claim.

Runtime shape¶

Run the API and worker as separate processes against one service-owned PostgreSQL database. The API creates/inspects jobs, serves snapshots, and, when explicitly enabled, accepts ordered Shopper interactions and performs bounded personalization. The worker streams each configured source inside a consistent relational transaction and also performs bounded personalization expiry and deletion reconciliation. Raw source interaction rows are never exported or staged; bounded group state is reduced into ephemeral derived aggregates using DuckDB (default) or Polars.

Select the complete aggregation engine in the deployment JSON:

{
  "pipeline": {"aggregation_backend": "polars"}
}

Use "duckdb" or omit aggregation_backend to retain the default. Polars computes co-occurrence, popularity, behavioral scoring, and evaluation aggregates over temporary derived-only Parquet; source adapters, metadata similarity, publication, and serving keep their existing contracts. See training backend operations for the different thread and memory controls.

Python 3.12–3.14 is supported. The supported local bootstrap uses uv and expects the sibling ../telemetry checkout pinned by the repository workflow.

make setup
make doctor
export RECOMMENDATIONS_CONTROL_DATABASE_URL='postgresql+psycopg://...'
export RECOMMENDATIONS_DATA_SOURCES_FILE="$PWD/config/data_sources.json"
export MERCHANT_PLATFORM_DATABASE_URL='postgresql+psycopg://...'
uv run --frozen alembic upgrade head
uv run --frozen recommendations-api

In another process:

uv run --frozen recommendations-worker

Observability runtime¶

Application observability is disabled by default and the base installation does not import or require OpenTelemetry. The reusable runtime and Collector assets live in the sibling telemetry repository; this project retains the recommendation-owned metric/event schema and domain adapter. The backend-neutral dashboards, alerts, fixtures, and response procedures are defined in operations/recommendations.toml and docs/recommendations-operations.md. Install .[observability,postgresql] to enable the process-owned runtime, then configure the strict observability group shown in config/data_sources.example.json. API and worker processes send enabled traces, metrics, and structured logs to one OTLP Collector endpoint; vendor/backend selection and credentials remain Collector-only concerns.

Non-secret precedence is code defaults, deployment JSON, then the allowlisted RECOMMENDATIONS_OBSERVABILITY_ENABLED, RECOMMENDATIONS_DEPLOYMENT_ENVIRONMENT, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL, and OTEL_EXPORTER_OTLP_CERTIFICATE overrides. OTLP header and client-key values must use environment or file references in JSON. Secret values are resolved only during runtime construction and are excluded from configuration fingerprints, representations, errors, logs, and metrics. Insecure transport is accepted only for local or test; enabled production-like environments require TLS. Configuration changes take effect on process restart, not hot reload.

For local development, uv resolves telemetry from ../telemetry through the editable source in pyproject.toml. CI checks out a pinned telemetry commit at that sibling path so locked resolution is reproducible. A published distribution declares the compatible telemetry version range and must resolve an independently published telemetry wheel rather than the local source override.

Commerce Simulation Harness with Docker Compose¶

Build the shared image and start the complete local simulation stack—separate control and simulation PostgreSQL databases, migrations and seed initialization, API, worker, and browsable storefront—with:

docker compose up --build

The image consumes the sibling telemetry checkout as a named build context. A direct build uses the equivalent command docker build --build-context telemetry=../telemetry ..

The API is available at http://127.0.0.1:8000 and the Simulation Storefront at http://127.0.0.1:8080. Compose mounts config/simulation-data-source.json read-only, initializes a fixed local/test-only Commerce Scope, and uses named volumes for both databases, worker scratch, and authoritative evidence reports. The complete workflow and safety boundary are documented in docs/commerce-simulation-harness.md.

Run the default deterministic independent-behavior journey through the same storefront HTTP actions available to a human browser with:

docker compose --profile runner run --rm simulation-runner

The resulting evidence describes only the synthetic scenario. It is not evidence of real business outcomes, causal uplift, production capacity, or production scalability.

To use Compose with a separate merchant-like source instead of the simulation defaults, copy the example source configuration, set RECOMMENDATIONS_DATA_SOURCES_FILE_HOST, and expose the URL named by that file's connection_url_env setting. For example:

Before submitting a Training Run, point the example source configuration at a relational source that contains the configured tables:

export RECOMMENDATIONS_DATA_SOURCES_FILE_HOST=./config/data_sources.json
export MERCHANT_PLATFORM_DATABASE_URL='postgresql+psycopg://user:password@host/source'
docker compose up --build

Override POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_PORT, SIMULATION_POSTGRES_PORT, RECOMMENDATIONS_API_PORT, or SIMULATION_STOREFRONT_PORT as needed. The Compose defaults are intentionally restricted to local/test environments and are not production credentials or orchestration.

Copy config/data_sources.example.json and adapt its three required ordered canonical queries. Each source must also list its allowed tracking_id and catalog_id pairs under scopes; both training and serving reject any other boundary. Catalog queries must return and filter both identifiers. purchases_query supplies online purchases; offline_purchases_query may be omitted when the merchandiser does not provide in-store purchase data. compatibility_query is also optional and streams configured product-to-product complement rules ordered by anchor and candidate. Root-level freshness settings support per-property overrides, while retention controls snapshot, run, and failed-diagnostic lifetimes. The optional root-level quality_drift group is disabled by default; when enabled, it controls the bounded history window, sample minimums, material-change floors, and robust-deviation thresholds. A source dialect is supported only after its driver and the source contract suite pass. Install .[snowflake] for Snowflake; any SQLAlchemy dialect can use the same contract. SQLite is intended for local/focused execution, not the production concurrency claim.

Generated Relational Data Source¶

To learn how the generator constructs Catalogs, shopping contexts, and reproducible evidence, read the synthetic data generation walkthrough, including Mermaid diagrams and a hands-on smoke-profile exercise.

Deterministic synthetic source generation is documented in docs/generated-relational-data-source.md. Generate a verified SQLite smoke source and its lineage artifacts with:

recommendations-generate-source --profile smoke --output-dir artifacts/generated-smoke

The source uses the same canonical adapter boundary as a merchant source. Its generation and materialization manifests contain logical digests and bounded oracle observations; raw generated interaction rows are not part of the evidence documents.

To stream the generated source directly into a PostgreSQL database, install the postgresql extra, place the SQLAlchemy connection URL in an environment variable, and name that variable without exposing the credential in process arguments:

recommendations-generate-source \
  --profile development \
  --database-url-env GENERATED_SOURCE_DATABASE_URL \
  --workers 4 \
  --output-dir artifacts/generated-development

The destination must not already contain the five generated-source tables unless --replace is explicitly supplied. PostgreSQL uses bounded, concurrent COPY workers plus parallel index and receipt verification. They write uniquely named staging tables; one final transaction publishes the fully verified tables, so failure cannot replace the last good source. --workers defaults to 4 and accepts 1 through 16.

For a dedicated local PostgreSQL container and one-shot generator, use compose.generated-source.yaml. The setup commands and local password configuration are in the generated source guide.

To retain generated history and add activity through a newer cutoff, use PostgreSQL --append with the existing output directory. It inherits the source identity and adds rows at the base profile's daily rate, with atomic publication and duplicate-safe exact-cutoff retries. See the append commands and limits.

API¶

Submit asynchronous training:

curl -X POST http://127.0.0.1:8000/v1/training-runs \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: daily-2026-08-05' \
  -d '{"data_source_id":"merchant-platform","tracking_id":"store-42","catalog_id":"main"}'

Inspect the returned run ID:

curl http://127.0.0.1:8000/v1/training-runs/RUN_ID

Training status includes configuration/model versions and the ordered pending, running, and terminal state transitions with UTC timestamps. The configuration version is a canonical SHA3-512 identity of source mappings and training policy; scoring grids, trend weights, and freshness policy therefore remain reproducible across workers.

Serve an anchored lane:

curl 'http://127.0.0.1:8000/v1/catalogs/main/items/SKU-1/recommendations/frequently-bought-together?data_source_id=merchant-platform&tracking_id=store-42&limit=20'

Global lanes use /v1/catalogs/{catalog_id}/recommendations/{best-sellers|most-viewed|trending}. Serving never queries a merchant database and continues to return an older snapshot with is_stale: true when a newer run fails.

Shopper personalization¶

Shopper personalization is additive and disabled by default. Enabled Commerce Properties can append ordered, idempotent interactions through POST /v1/personalization/interactions, request in-lane reranking with personalize=true, or request the global for-you strategy. Authentication is derived from the bearer principal; signed Personalization-Context and causal tokens bind every operation to one data source, property, Catalog, and opaque Shopper ID. Personalized serving reads only the current Recommendation Snapshot and its feature companion—it never queries the merchant source. See docs/shopper-personalization.md for the complete contract, privacy lifecycle, evaluation boundary, and rollout gates.

Verification¶

make setup
make doctor
make test
make smoke

Use make test-focused TEST=tests/path.py::test_name for the shortest feedback loop. The complete command ladder, PostgreSQL boundary, generated-artifact rules, and Apple Silicon macOS CI contract are documented in docs/testing.md.

Migrations can be round-tripped on a disposable database with alembic upgrade head and alembic downgrade base. Export the HTTP schema with:

uv run --frozen python scripts/export_openapi.py artifacts/openapi.json

The 200,000-product / 100M-view / 100M-purchase and latency objectives require a preloaded relational qualification source and the approved reference worker; they cannot be established by the focused local suite. Run the retained driver against that environment:

uv run --frozen python scripts/qualify.py \
  --base-url http://service:8000 \
  --data-source-id qualification \
  --tracking-id qualification-store \
  --catalog-id qualification-catalog \
  --anchor-product-id SKU-000001 \
  --artifact artifacts/qualification.json

The command fails unless the recorded source counts, 12-hour batch objective, 500 ms submission objective, and 90/150 ms serving percentiles all pass. Monthly availability remains an operations measurement: use successful eligible requests divided by all eligible requests, excluding planned maintenance only if the service-level agreement explicitly permits that exclusion.

Operational behavior¶

One active run is database-enforced per (data_source_id, tracking_id, catalog_id).
Workers renew leases while training. A lost run is reclaimed once; a second loss is terminal.
Snapshot rows are loaded in bounded chunks while invisible. Publication verifies the complete strategy manifest, swaps the serving head atomically, then applies retention.
Five successful snapshots are retained per scope by default. Training metadata is retained for 90 days and bounded failure diagnostics for 30 days.
Browsing Sessions and Orders above their configured distinct-product limits are excluded whole from co-occurrence; the limits default to 100 and 200 respectively, and excluded rows still contribute to popularity and exclusion metrics.
Successful Training Runs persist versioned aggregate evidence. Quality/drift assessment compares only the latest same-scope runs with matching model, configuration, and Training Evidence Schema versions; it uses median/MAD baselines and never changes snapshot publication or serving state.
The Training Evidence Schema version is independent of the OpenTelemetry signal schema. Missing additive evidence is reported as unavailable, and crossing a drift threshold is diagnostic—not a claim about clicks, conversion, revenue, retention, attribution, or causal performance.