Make checkpoint lifecycle transitions explicit

Status: Implemented in the local working tree and verified with the complete local gate. Test interfaces follow the existing contracts, with direct application/group tests as proposed to the user. This engineering specification preserves the approved commerce contracts.

Problem Statement

Checkpoint execution mixes admission draining, idempotent submission, polling, verification, lifecycle callbacks, and repeated optional-lifecycle conditions. Failure and cancellation ordering are difficult to trace.

Solution

Use a Null Object for absent lifecycle integration and cohesive transition operations that keep checkpoint execution readable while preserving durable ordering and fail-closed behavior.

User Stories

1. As a developer, I want an always-present lifecycle interface, so that optional checks do not obscure transition logic.
2. As a developer, I want named outcome transition operations, so that timeout, cancellation, verification, and failure have explicit owners.
3. As a operator, I want completed checkpoint replay remains idempotent, so that retries do not submit duplicate training.
4. As a operator, I want timeout retains the known Training Run, so that retrying never substitutes a different run.
5. As a operator, I want admission closes and drains before evidence capture, so that checkpoint source boundaries remain valid.
6. As a operator, I want cancellation remains durably ordered, so that the lifecycle cannot complete before cancellation is recorded.
7. As a operator, I want snapshot and source verification required before reopening, so that traffic cannot resume on inconsistent evidence.
8. As a operator, I want intermediate success alone permits reopening, so that final, canceled, or failed runs stay closed.
9. As a tester, I want fake gateways and controlled time remain supported, so that tests remain deterministic and fast.
10. As a maintainer, I want same error categories and durable records, so that existing reports and recovery behavior remain compatible.

Implementation Decisions

• Implement an explicit no-op CheckpointLifecycle adapter and resolve the optional dependency once at construction.
• Extract preparation, terminal poll handling, verification/completion, and classified failure transitions without creating an unnecessary State class hierarchy.
• Keep durable record writes, cancellation signaling, lifecycle completion/failure, and admission changes in their existing semantic order.
• Preserve public execute arguments, idempotency keys, polling/backoff bounds, recovery records, and exception classification.
• Keep Training and Serving API gateways as the only recommendation-system access seam.

Testing Decisions

• Use existing CheckpointOrchestrator tests with fake time and gateways, including replay, cancellation ordering, failed training, source evidence failures, and snapshot mismatch.
• Add parity coverage for absent versus recording lifecycle adapters and any transition path currently missing an observable assertion.
• Run checkpoint/lifecycle tests, simulation integration tests, static checks, the full suite, and installed-wheel verification.
• Successful checks alone are insufficient: review the final implementation against every story, verify ownership moved into cohesive objects, and confirm duplicated decisions were removed.
• Keep explicit validation guards when they communicate invariants; do not target an arbitrary count of conditional statements or introduce classes that merely forward calls.

Out of Scope

New checkpoint states, worker cancellation, retry policy changes, direct recommendation database access, and simulation traffic changes.

Further Notes

Preserve Commerce Scope isolation, snapshot-only serving, atomic publication, bounded resources, and exclusion of credentials and raw interaction identities from retained diagnostics. The refactor is complete only after implementation, focused verification, repository gates, and code-aligned documentation updates. PostgreSQL checks require an explicitly disposable database; record their omission if none is available. No deployment or hosted-setting change is authorized.