DuckDB vs Polars aggregation benchmark (2026-09-25)

This measures the repository's complete derived aggregation work-store ingestion path, from work-store startup through generated view reduction and writer completion. Verification queries and cleanup are outside the timed interval. It does not include source I/O, purchases, candidate scoring, snapshot publication, or serving. The generated views are deterministic and contain no merchant records.

Environment: Apple M2 Pro (arm64), macOS, Python 3.14.7, DuckDB 1.5.5, Polars 1.44.0. Each trial ran in a fresh process, sequentially, with four configured threads, a four-thread Polars pool, and a batch_key_limit of 100,000. Both engines passed the benchmark's analytic checks for pair, support, popularity, and group totals.

Workload	DuckDB wall seconds	Polars wall seconds	Result
50,000 groups × 20 items (1,000,000 rows)	4.438587, 4.500547, 4.570147	3.346787, 3.232779, 3.206948	Polars median 3.233 s vs DuckDB 4.501 s (1.39× faster)
100,000 groups × 3 items (300,000 rows)	0.691481	0.760600	DuckDB 10% faster in this single trial
2,000 groups × 20 items (40,000 rows)	0.211272	0.301754	DuckDB 43% faster in this single trial

For the million-row case, median process CPU time was 7.513 seconds for DuckDB and 4.422 seconds for Polars. These are local development measurements, not production capacity or an end-to-end Training Run qualification. The two smaller cases have one trial each and should not be treated as stable estimates. This workload uses 10,000 possible product IDs, so results can change with catalog cardinality, group overlap, batch size, and available memory.

Reproduce with make benchmark-training, using the same arguments for each engine:

make benchmark-training BENCHMARK_ARGS='--aggregation-backend duckdb --groups 50000 --group-size 20 --threads 4 --batch-key-limit 100000'
POLARS_MAX_THREADS=4 make benchmark-training BENCHMARK_ARGS='--aggregation-backend polars --groups 50000 --group-size 20 --threads 4 --batch-key-limit 100000'

Run each command three times in alternating engine order for the million-row comparison. Change --groups and --group-size together to reproduce the other cases. The command prints wall time, process CPU time, throughput, and the aggregate-total verification result as JSON.