Co-occurrence backend benchmark suite

make benchmark-cooccurrence-suite compares the six implementations of group co-occurrence counting on the real DerivedWorkStore path. All generated view rows belong to one synthetic Commerce Scope and are streamed one group at a time. The suite does not read a merchant source, retain raw rows, publish snapshots, or include metadata similarity and serving work. It is a reduction benchmark, not a full Training Run qualification claim.

Run it

make benchmark-cooccurrence-suite SUITE_ARGS='--profile smoke --backends duckdb sparse --threads 4 --memory-mb 256 --temp-gb 8 --disk-cap-gb 16 --batch-key-limit 100000 --repetitions 1 --warmups 0 --timeout-seconds 1800 --power off --output .build/cooccurrence-smoke.json'
make benchmark-cooccurrence-suite SUITE_ARGS='--profile quick --backends duckdb sparse --threads 4 --memory-mb 256 --temp-gb 8 --disk-cap-gb 16 --batch-key-limit 100000 --repetitions 1 --warmups 0 --timeout-seconds 1800 --power off --output .build/cooccurrence-quick.json'
make benchmark-cooccurrence-suite SUITE_ARGS='--profile matrix --backends duckdb sparse --threads 4 --memory-mb 256 --temp-gb 8 --disk-cap-gb 16 --batch-key-limit 100000 --repetitions 1 --warmups 0 --timeout-seconds 1800 --power off --output .build/cooccurrence-matrix.json'
make benchmark-cooccurrence-suite SUITE_ARGS='--profile out-of-core --backends duckdb sparse --threads 4 --memory-mb 256 --temp-gb 8 --disk-cap-gb 16 --batch-key-limit 100000 --repetitions 1 --warmups 0 --timeout-seconds 1800 --power off --output .build/cooccurrence-out-of-core.json'

These commands use the same two backends, thread count, resource limits, batch threshold, repetition count, timeout, and power mode. Only the profile and report path change. They give a consistent exploratory baseline; one measured trial cannot estimate run-to-run variation. The four quick cases also appear unchanged in matrix, so those results can be compared directly. smoke and out-of-core use different cases and answer different workload questions; compare backends within each case rather than ranking times across profiles. For more stable latency estimates, rerun comparable profiles with the same --repetitions 3 --warmups 1 settings. For a thread-scaling sweep, use the same --threads 1 4 8 list in every profile being compared. A six-backend sweep likewise needs the same --backends duckdb sparse accelerate torch mlx ane selection in each profile, subject to backend availability and resource limits.

smoke checks two tiny patterns. quick varies group width (3, 20, 100), overlap, and repeated group multiplicity. matrix adds a dense small Catalog and two disjoint sparse Catalogs. The out-of-core profile contains two disjoint-product cases: 200,000 groups of 20 and 8,000 groups of 100. Their distinct pair-count values alone occupy at least 304 MB and 316.8 MB respectively, exceeding the default 256 MB DuckDB memory budget before keys and table overhead. This is an enforced database-memory-budget claim, not a claim that either workload exceeds physical RAM. The worker still uses bounded batches, but Python/Arrow allocations lie outside DuckDB's memory_limit; peak process RSS is reported independently. Query spill is measured separately from the persistent work-store file and can legitimately be zero even when the derived data exceeds the configured memory budget.

The default suite runs three measured repetitions and one warm-up per case/backend/thread in fresh processes; out-of-core defaults to one measured repetition and no warm-up due to cost. Backend order rotates between repetitions. Use --case NAME to select a case, --backends and --threads to select comparison dimensions, and --repetitions/--warmups to increase evidence. --batch-key-limit controls the existing bounded summary flush threshold; keep it identical between backend comparisons. An unavailable optional backend is recorded rather than silently substituted; --require-all turns unavailability into a failing suite. --timeout-seconds bounds each trial. Every trial gets an isolated temporary scratch root that is cleaned even after a timeout. The suite checks free space before the out-of-core profile and stops streaming when the sampled scratch footprint exceeds --disk-cap-gb. DuckDB's own spill cap is set separately with --temp-gb.

Measurements and interpretation

The JSON report contains every observation, including failures, plus median, nearest-rank p95, minimum, and maximum ingest latency per case/backend/thread. Ingestion timing includes work-store startup, backend import/compilation, source reduction, and all background writer completion. It excludes verification queries and cleanup, which are tracked separately. The benchmark verifies pair and support totals against an analytic oracle, then compares two order-independent hashes of aggregated counts across successful backends. For disjoint-product cases, each key is unique by construction, so verification scans the keys directly instead of building another large grouping. Verification uses DuckDB SQL and does not materialize all pairs into Python. Hash agreement is strong evidence but not a formal proof of equal tables. Work-store disk and spill peaks are sampled every 250 ms, so brief peaks may be missed. Peak RSS is the operating system's process high-water mark. CPU seconds and average cores cover the full process, not GPU or Neural Engine execution. Page-fault and filesystem block counts are reported as operating-system counters, not byte counts.

On an already privileged macOS process, --power auto reads /usr/bin/powermetrics for CPU, GPU, ANE, and combined rail estimates. The collector does not invoke sudo or request privilege. Without access, or when a trial finishes before a usable sample, power is explicitly unavailable. --power required fails in that situation; --power off avoids measurement overhead. Reported watts and estimated joules are system-level approximations over the timed ingest interval, not power attributable solely to the worker. Other machine activity, sampling overhead, battery state, and thermal conditions can distort comparisons; do not add the combined rail to its components or compare estimated watts across different machines. For small cases, disable power and use the longer profiles for energy analysis.

Run on an otherwise quiet machine, retain the exact JSON and installed dependency versions, and compare like-shaped cases. The suite is not a substitute for an end-to-end Training Run benchmark: source validation/I/O, model selection, Catalog metadata processing, and snapshot publication are outside its measurement window.