Serving load implementation handoff

Objective and state

• Objective: implement the user-approved Locust serving-only disposable local environment, reproducible workload, and performance/CI evidence contracts.
• Status: implemented and ready for review. The completed local reference fails the approved latency gate; its throughput, correctness, completeness, and cleanup checks pass.
• Governing specification: 58-story bundle, Linear PL-261.
• Boundary: development tooling and synthetic serving experiments. Production HTTP contracts, source/serving isolation, telemetry ownership, and released generation profiles are preserved.

Working tree

• Base: main, 6de3ace728ee6909f355210e0b0735a2760d4a67.
• Changed: optional dependency/lock, Docker stages/owned Compose, Make and manual workflow, workload/measurement/evidence package, outer preparation/controller/Locust/runtime-smoke scripts, focused tests, and governing implementation/operation/spec documentation.
• Preserved prior work: agreed load specs and index navigation; graphify query memory and stamp.
• Generated: ignored .build/, load evidence under artifacts/load-testing/, owned disposable Docker resources, and automatically regenerated tracked setuptools metadata. No raw source data or private oracle is copied into retained evidence.

Verification evidence

All Python commands used PyCharm's configured SDK preflight. Existing-environment checks used UV_NO_SYNC=1 and the configured UV_PYTHON; isolated runtime checks selected their explicit supported Python version. These environment controls are omitted from the table for readability.

Command	Result
make test-focused TEST=tests/delivery/test_repository_harness.py	Baseline: 5 passed
Focused new policy tests before package creation	Red: missing recommendations.load_testing
make test-focused TEST=tests/unit/test_load_testing.py	12 passed
make test-focused TEST=tests/unit/test_load_lifecycle.py	4 passed
make test-focused TEST=tests/unit/test_load_lifecycle.py::test_late_storage_failure_cannot_leave_a_passing_summary	Red: completion-marker write failure left a passing summary; moved the final summary after required marker/retention writes
Final make test-focused TEST=tests/unit/test_load_lifecycle.py	6 passed, including marker and retention failure cases; neither leaves a passing summary
make test-focused TEST=tests/contract/test_load_serving.py	6 passed after eligibility/category additions
make load-test-runtime	Red reproduced 100/100 synchronized starts in one 40 ms slice; green after phase offsets, including closed-loop wait assertions
make load-test-runtime PYTHON_VERSION=3.12	Passed real HTTP/events, timeout/sanitization, bounded UI, phase/pacing checks on Python 3.12.10/macOS ARM64
make load-test-runtime PYTHON_VERSION=3.13	Same checks passed on Python 3.13.15/macOS ARM64
make load-test-runtime PYTHON_VERSION=3.14	Same checks passed on Python 3.14.7/macOS ARM64; dashboard startup/shutdown race reproduced and fixed
make compose-check	All three Compose configurations passed
uv run --frozen mypy scripts/load_testing/run.py scripts/load_testing/prepare.py src/recommendations/load_testing	6 source files passed
uv run --frozen mypy load_testing/driver.py	Strict driver check passed after typed event context and narrow third-party boundaries
make test	Static passed; 838 unit passed/6 optional skipped; 139 contract/delivery passed; 140 integration passed/36 PostgreSQL deselected
make smoke	Fresh source archive/wheel and clean installed-package report passed; ordinary install contains no Locust
make doctor	Supported SDK, locked dependencies, both sibling checkouts, tools, and Docker checks passed
Final make test-contract	141 contract/delivery tests passed
Final make docs-check	All 72 maintained Markdown files passed link/command validation
make test-static after final tooling changes	Ruff, 79 package source files, 2 outer scripts, 72 documentation files, and architecture passed
make test-focused TEST=tests/delivery/test_serving_load_workflow.py	2 passed, including manual rollout, preserved ownership reports, and recovery before a new invocation
Final make load-test-runtime PYTHON_VERSION=3.12, 3.13, and 3.14	7 actual HTTP events each, including malformed 429 rejection and distinct policy-unavailable 503; UI pages, sanitization, timeout, phases, and pacing passed
Configured SDK Python /tmp/recommendations-load-interruption-probe.py	Actual SIGTERM during preparation: exit 1, owned resources/scratch removed, separate sentinel container preserved
Same probe with --force-recovery	Actual SIGKILL followed by controller --cleanup-active: interrupted evidence stays failed, owned resources/scratch removed, sentinel preserved
graphify update . after each code completion	Final AST update passed: 7,126 nodes, 16,187 edges, 254 communities; preserved existing curated nodes and refreshed affected hub labels; no semantic/API extraction
git diff --check and final labelled Docker inventory queries	Clean whitespace; no owned load containers, volumes, networks, or verification sentinels remain

Formatting/import corrections used scoped ruff format and ruff check --fix on the new tooling and tests. Initial lint failures were long lines/import order and a documented cancellation raise; these were corrected. Mypy first caught recovery report inference and untyped Locust callback boundaries; strict checks subsequently passed.

Disposable runs

Each engineering rerun explicitly invoked make load-test, retained its own outcome, and attempted verified teardown. There are no retries in the controller or HTTP workload.

Run	Result and diagnosis
815d2c9f552249caa7430646627d63fe	Preflight failed; sibling admission checkout has no Git HEAD; cleaned
cba51ccf1c5340dd8e0b5fd9049d5800	Same prerequisite diagnosis; record optional revision plus source digest; cleaned
f070e944c1524044ac659e7b59b2882b	Publication startup failed; migration Compose invocation stopped its database; cleaned
b75da4e2b361484d9f4cb6c6d8393c24	PostgreSQL volume initialization failed under capability restrictions; run official image as postgres; cleaned
34dd13731d3047d989655c1af1281a8d	Complete reference; 74,962 valid measured responses, 249.8733 valid RPS, zero request/semantic/admission errors, minimum bucket samples 1,529; failed latency correctly; cleaned
8463a98ad57a46edbf262059a1020ed1	Complete reference after dephasing and independent eligibility/category checks: 74,000 valid responses, 246.6667 valid RPS, zero errors/admission, minimum bucket samples 1,515; failed latency correctly; cleaned
7542b2b31b64485883a1e1ea2cc43dd8	SIGTERM preparation probe: failed outcome and complete cleanup; unrelated sentinel preserved
23699a322aa04d9ba5928a080fbf1b40	SIGKILL preparation probe plus active-run recovery: failed outcome and complete cleanup; unrelated sentinel preserved

The first complete run's approximate p50/p95/p99 were 150/290/330 ms. Original counters were 13,859 at or below 90 ms and 40,132 at or below 150 ms out of 74,962. The gate did not pass. Generator CPU had headroom. A minimized deterministic probe then demonstrated synchronized pacing. The second complete reference tests the correction without changing the approved gate. Its approximate p50/p95/p99 were 250/310/350 ms; original counters were 375 at or below 90 ms and 1,125 at or below 150 ms out of 74,000. Latency still fails overall and in every bucket. Dephasing fixes the demonstrated generator burst; it does not resolve the measured serving latency.

The second run took 1,393.68 s overall. It recorded 48 host/container samples, generator peak CPU 8.8% and RSS 276,905,984 bytes, peak sampled API CPU 141.29% of a 200% allocation, control CPU 56.61%, and at most 16 control-database connections. No saturation, lost-user, lifecycle, or semantic invalidity was detected. Docker CPU throttling and host memory observations are explicitly unavailable. These observations do not identify a definitive serving bottleneck.

Actual container runtime was Linux ARM64 Python 3.14.8, Locust 2.46.6, gevent 26.9.0, and geventhttpclient 2.5.1. Both complete runs used all four full-size fixture scopes and all 44 buckets; source/preparation were stopped before HTTP readiness. JSON/CSV reports remain under each listed run's ignored artifact directory. No passing reference or CI-runner qualification is claimed.

Final reporting commits the summary only after the other required evidence writes and retention. A late marker/retention failure was reproduced deterministically and fixed after the measured runs; it changes finalization order and no workload, gate, or serving behavior. Focused lifecycle and static checks were rerun; another full performance experiment was unnecessary for this change.

Decisions and risk

• See the design record and story evidence matrix.
• Local sibling telemetry is available and included as a build context. Required resource observations use Docker/psutil; no sibling telemetry changes or new direct imports are needed.
• Reference traffic, phase controls, and gate arithmetic have separate evidence. Capacity/sustained timing and CI-runner qualification are not inferred from deterministic phase checks.
• No migrations changed. Separate PostgreSQL-marked regression suite has not been run; the load experiment itself uses owned disposable PostgreSQL 18 for generation/publication/serving.
• Full stepped/sustained wall-clock runs and hosted workflow dispatch have not been performed. Their controls are smoke checked; no capacity or sustained qualification is claimed.
• Full Linux Python 3.12/3.13 runtime matrix and documentation-site rendering were not run. The actual generator uses Linux 3.14; supported macOS versions and repository documentation gates are checked. No website deployment is part of this change.
• Manual CI remains pending the runner's valid baseline before pull-request promotion.
• make verify was not invoked: its constituent make test and make smoke gates were run, followed by focused checks for the final controller/workflow changes. No migration change requires a separate migration gate; normal integration already ran migration round trips.
• Remaining qualification: diagnose serving latency under this envelope before claiming a passing reference; run full stepped/sustained experiments when measuring those profiles; establish a valid CI-runner baseline before automatic pull-request promotion.

Authority

• User authorized implementation and disposable local execution; use of sibling telemetry is explicit.
• No commit, package publication, deployment, hosted setting changes, or tracker administration is part of this turn. No pending human requirement decision blocks implementation.

2026-10-02 CI preflight remediation

The repeated CI failure at 991ec2aa66f54cf347ff4b81b7fb7ff8a19ff21f occurred before serving traffic. Run ad3b5e21181b43e6b4bf6374d877da09 retained preflight / command_failed, empty measurement windows, and successful cleanup. Its old diagnostics did not identify the command. The installed Actions LaunchAgent has SessionCreate=true, while the personal Docker client selects the desktop credential store. Read-only inspection excluded credentials and raw runner environment values.

Interactive preflight passed both from the development checkout and from the runner checkout with CI=true. A temporary LaunchAgent using the runner's PATH and SessionCreate=true reproduced Docker info/Compose success followed by docker pull --quiet postgres:18-alpine exit 1, classified from stderr as credential lookup failure. Only that classification was displayed or retained. Replacing the client configuration in the same isolated session made the pull and public Buildx metadata lookup pass. This identifies the security-session-dependent credential lookup as the reproduced prerequisite failure.

The workflow now prepares a private anonymous client configuration under RUNNER_TEMP before both recovery and execution. Only context selection, plugin discovery, and a reference to the existing context store are preserved. Personal registry credentials/settings are not changed or copied. The explicit credential-free Hub entry matters: Docker's configuration loader otherwise auto-discovers a native credential store when authentication configuration is empty. The runner temporary directory is managed by the Actions runner. Anonymous public registry access remains subject to ordinary network and rate-limit failures.

Controller failures now expose finite operation names and observed exit codes in the console and the additive diagnostics.command_failures field. Existing report version, failure codes, measurement gates, and cleanup rules are preserved. No raw command arguments/output are retained.

Command or bounded probe	Result
git status --short, git rev-parse HEAD	Base matched the supplied CI log; initial .vocab.txt preserved
graphify query 'serving load test preflight failure diagnostics docker compose k6' --budget 2000	Located the controller and preflight boundary; confirmed against source
Configured-SDK inline preflight() probes, development and runner checkout	Both passed interactively; CI=true alone did not reproduce the problem
Direct child SessionCreate(0, 0) probe	Unsupported status 100001; replaced by a temporary LaunchAgent matching the actual runner configuration
Temporary launchctl bootstrap / bootout, original Docker configuration	Reproduced credential lookup failure at PostgreSQL pull; Docker info and Compose passed; temporary job removed
Same LaunchAgent, anonymous client configuration	PostgreSQL pull and docker buildx imagetools inspect python:3.14-slim passed; temporary job removed
Same LaunchAgent, actual docker_client.py plus full preflight(), public-image build, and owned cleanup	All passed; build used Dockerfile frontend, python:3.14-slim, and ghcr.io/astral-sh/uv:0.12.5; owned image and temporary job/configuration removed
make test-focused TEST=tests/unit/test_load_lifecycle.py::test_preflight_command_failure_identifies_operation_without_raw_output	Red: all five preflight command failures lacked operation/exit evidence
make test-focused TEST=tests/unit/test_load_lifecycle.py	Green: 11 passed, including raw-output exclusion and cleanup for every preflight command
make test-focused TEST=tests/unit/test_load_docker_client.py	7 passed: anonymous configuration, preserved context/plugins, unchanged personal settings, default daemon, overwrite refusal, and safe invalid-config failures
make test-focused TEST=tests/delivery/test_serving_load_workflow.py	4 passed, including configuration propagation before recovery/execution
Initial make test	Ruff found two long lines; corrected
Second make test	Static gates and all 858 unit tests passed; contract/delivery found a stale expected lifecycle-test count (6 instead of 11); updated
make test-contract test-integration	145 contract/delivery and 140 integration tests passed; 36 PostgreSQL cases deselected
Final make test-static	Ruff, strict mypy for 79 package files and three outer scripts, 73 Markdown documents, and architecture passed
graphify update .	AST refresh succeeded: 7,181 nodes, 16,276 edges, 225 communities; retained 216 existing nodes from one file outside the current scan rather than pruning them; no semantic/API extraction
git diff --check	Passed
gh run list, gh run view 37068010563 --json jobs, owned Docker inventory	New run on aeddc55 passed client configuration/recovery and advanced beyond preflight/build; both PostgreSQL containers healthy and preparation running; performance outcome still pending

All Python checks used the configured SDK; Make checks used UV_NO_SYNC=1 and UV_PYTHON=/Users/pierre-luc-delisle/PycharmProjects/recommendations/.venv/bin/python. The normal gates are completed through the commands above and the final static check. The preflight/build probe validates the corrected failure boundary. The separately started CI run 37068010563 also advanced to preparation with healthy source/control PostgreSQL containers. Its serving measurement and final cleanup outcome remain pending; this change makes no new performance or CI-baseline claim. make smoke, client runtime compatibility, migration, Compose-artifact, and PostgreSQL gates were not repeated because packaging, workload, migrations, Compose definitions, and database behavior did not change. This agent performed no commit, push, deployment, or hosted settings change. During verification, concurrent work committed the implementation as aeddc55; local main and origin/main both point to that commit. The new CI run uses that revision. GitHub job reruns preserve the original commit, so a rerun of the original 991ec2a job would still lack the fix. Unrelated research and graph query/reflection changes that appeared during execution were preserved.