Recommendations operations

operations/recommendations.toml is the backend-neutral dashboard, alert, and runbook contract. Deployments translate its intent into their monitoring backend without adding request, scope, product, run, snapshot, session, order, account, tenant, or user identifiers to metric dimensions. Those identifiers remain trace/log correlation fields and durable application evidence.

Every investigation starts by recording the deployed application revision, telemetry wheel SHA3-512, dependency lock SHA3-512, OCI Collector image digest and configuration SHA3-512, configuration fingerprint, process role, deployment environment, and UTC window. Preserve query exports or screenshots with the incident; never paste credentials, request bodies, recommendation payloads, or raw SQL inputs into telemetry.

First-consumer release evidence is assembled with scripts/build_telemetry_evidence.py. Each of its six JSON gate inputs must report pass and the SHA3-512 of the same candidate wheel. The command checksums those inputs, the dependency lock, and Collector configuration and refuses incomplete, failed, skipped, or stale evidence. The resulting manifest is then validated by the telemetry repository's scripts/verify_consumer_evidence.py; the builder does not execute or invent hosted qualification results.

Serving errors or latency

1. Split failures and latency by route template, method, status class, and bounded strategy. Never query by a raw URL or product/catalog identifier.
2. Compare application request counts with ingress counts. A mismatch indicates routing or instrumentation trouble rather than model behavior.
3. Use one sampled request trace to separate admission, control-database, snapshot lookup, and serialization time. Confirm the durable serving head and snapshot state directly.
4. If telemetry is incomplete, diagnose the Collector separately; serving remains authoritative and must stay fail-open with respect to telemetry.
5. Recover only after error and latency objectives hold for the alert recovery window.

Opt-in embedded ANN

The deployment document may include "ann": {"scopes": [{"data_source_id": "...", "tracking_id": "...", "catalog_id": "..."}], "dimensions": 512, "model": "metadata_svd"}. Set "model": "two_tower" only for an experimental PyTorch-trained representation. It needs the optional torch package in Training Workers (for a local full environment, run uv sync --locked --extra dev --extra postgresql --extra observability --extra torch); API processes need only the published NumPy weights and Faiss index. Two-tower training uses Apple MPS when available and CPU otherwise, capped at 128 latent dimensions, 10,000 aggregate positive pairs, three epochs and 128-pair batches. A scope with fewer than two usable pairs publishes no learned artifact and retains ordinary snapshot fallback. Omit ann or leave scopes empty for the default-off path. Each scope must already be configured as a Relational Data Source scope. The same document must reach workers and API processes; existing snapshots without an ANN artifact continue to serve through the old paths until a successful opted-in Training Run.

The optional artifact is stored in the control database, not in process-local files. Budget database storage and transfer for the versioned vector matrix plus serialized native HNSW index; the learned mode also stores input vectors and query weights. Construction rejects a generation above 1 GB of these bytes. API processes retain at most two loaded generations each, with no native index shared between processes. This is a hard artifact-size bound, not evidence that multiple scopes, overlapping generations, or full process RSS satisfy the proposed 2 GiB gate. An API process loads at most one cold artifact at a time in a background worker; requests use the ordinary Snapshot fallback until that load completes, so a new serving head need not block a request on database transfer or native deserialization.

Do not enable ANN broadly until the separate representation-relevance, same-vector recall, restrictive filter, full Serving API latency, memory, build/load and Linux compatibility gates in PL-244–247 pass. The current synthetic HNSW measurements do not pass the proposed recall gate. To roll back, remove the affected scope from ann.scopes in both API and worker configuration; the API immediately uses the old Snapshot paths, and subsequent Training Runs omit the artifact. A failed artifact build or publication leaves the previous serving head in place. An artifact read/validation failure falls back to old Snapshot candidates and emits a safe snapshot-ID warning without Shopper data.

Training aggregation backend

Set pipeline.aggregation_backend to "duckdb" (default) or "polars" in the deployment JSON and restart the worker. This selects all derived aggregation and scoring, independently of the merchant source dialect and service-owned snapshot database. Polars is included in the base installation. Leave cooccurrence_backend omitted/default with Polars; accelerator overrides remain available only for DuckDB aggregation.

For Polars, set POLARS_MAX_THREADS before process startup (for example POLARS_MAX_THREADS=4 uv run --frozen recommendations-worker). Its native pool is process-wide; pipeline.threads continues to configure metadata similarity but does not resize that pool. batch_key_limit bounds ingestion buffers, with the existing maximum-group overshoot. memory_limit checks the size of materialized derived batches and query results after allocation; it is not a hard cap on Polars query intermediates or process RSS. max_temp_directory_size checks owned derived Parquet storage after writes, footer closes, and compaction, allowing a transient write/compaction overshoot. Polars-internal spill is outside that owned-file budget. Use operating-system/container limits to enforce total memory and filesystem bounds.

Budget violations are classified as resource_exhausted; other computation failures retain the existing strategy_computation_error classification. Both leave the previous complete serving head intact. Temporary derived files are cleaned when the run exits. A process killed externally cannot perform Python cleanup; normal worker scratch-volume lifecycle policy still applies. Engine parity tests establish correctness on synthetic inputs, not faster training or production scale qualification. Switching back to "duckdb" affects subsequent Training Runs; existing snapshots remain servable without retraining.

Worker stall or lease loss

1. Confirm at least one worker emitted started and ready events and continues bounded polling.
2. Compare pending durable runs, active-run gauge, claim outcomes, and lease-renewal outcomes.
3. Pivot from a failed bounded outcome into the correlated run trace, then verify durable run and lease timestamps. Heartbeats intentionally do not create spans.
4. Do not manually publish a partial snapshot. Let lease reclamation and the database-enforced retry policy decide ownership; a second lease loss is terminal.
5. Recover after the run reaches a durable terminal state and representative renewals remain healthy.

Training failure or publication stall

1. Identify the first failed phase among source read, generation, evaluation, publication, quality assessment, and retention using bounded phase/outcome dimensions.
2. Check the durable bounded diagnostic and source/snapshot contracts. Raw source rows and generated recommendations must not appear in exported signals.
3. For publication, verify the complete strategy manifest and serving-head transaction. An older complete snapshot should remain servable when a newer run fails.
4. Retention failure is investigated after publication state is known; never delete snapshots solely to make a telemetry alert recover.
5. Recover after a representative run publishes a complete snapshot and the serving head is current.

Quality drift

1. Treat drift as diagnostic, not as evidence of clicks, conversion, revenue, retention, attribution, or causal performance.
2. Verify scope, model, configuration, metrics-schema versions, baseline run IDs, and minimum sample requirements from durable evidence before comparing values.
3. Review the bounded largest-deviation summary and the complete durable report. Do not add run or snapshot IDs to metric dimensions to simplify the query.
4. Publication and serving remain independent of the drift result. Escalate model/configuration changes through the normal review path.

Privacy or cardinality violation

1. Stop forwarding the affected signal at the Collector if that can be done without changing application behavior; retain the application and durable data plane.
2. Capture the registered metric/event name and attribute name, but do not copy the leaked value into tickets or logs. Rotate any exposed credential through its owning secret system.
3. Check the consumer schema and adapter mapping. Metric dimensions are restricted to the reviewed bounded vocabulary in the operations contract.
4. Re-run the privacy canaries, 100-context isolation test, and 10,000-identifier series test before re-enabling export.
5. Record backend deletion or retention actions and attach checksummed evidence to qualification.