Choosing a local analytics engine starts with a question about the work it needs to do. A fast grouped count does not establish how an engine handles a selective scan, a larger grouping state or a join. A benchmark also needs to establish that every engine answered the same question.
I built a comparison of Polars, DuckDB, chDB and pandas using 30,385,077 PyPI download events. The recorded CPU run completed 240 full-result checks across ten configurations and four workloads. This article explains the measurement contract and what the results support. GPU performance remains unverified.
Establish the result before measuring the engine
The input is the same set of 19 Parquet shards for every implementation. The original workload counts downloads by country and project, producing 463,523 groups. The comparison adds a country filter, a higher-cardinality aggregation by country and URL, and enrichment through a small unique project dimension.
| Workload | Operation | Result boundary |
|---|---|---|
| Original aggregation | Count by country and project | 463,523 groups |
| Selective filter | Select country GB, then count by project | 454,252 matching events |
| High-cardinality aggregation | Count by country and URL | 1,133,980 groups |
| Dimension join | Join project to a synthetic bucket, then count by country and bucket | 3,009 groups |
The dimension assigns each distinct non-null project a bucket derived from the first byte of its SHA-256 digest. It is a reproducible synthetic lookup, rather than a real project classification.
A separate streaming Python Counter implementation produces the reference results without importing a benchmark engine. Every warmup and measured output must match that reference in full. Null and empty keys remain distinct, null project keys do not join to the non-null dimension, and results use a deterministic count/key ordering. Only physical string and integer widths are normalized.
This catches disagreements that a matching row count or a handful of inspected groups would miss. Failed or incomplete configurations are excluded from successful comparison charts.
Give every trial a fresh process
The CPU harness separates the controller, reference generation, engine workers and verification. Each engine trial starts in a fresh interpreter, with the same 12-CPU affinity budget and explicit thread settings. One warmup precedes five measured repetitions. Rounds run serially in reproducibly shuffled order.
The host was an Intel Core i7-8750H with six physical cores, twelve logical CPUs and 30.5 GiB visible RAM, running Ubuntu 26.10 and Python 3.12.12. Relevant input bytes were pre-read before every phase. These are warm local-file measurements; memory pressure can still affect cache residency.
Query time includes query construction, file reads, execution, deterministic sorting and complete Arrow materialization. It excludes interpreter imports, reference generation, verification and result export. Connection opening is recorded separately. A complete command-line invocation will therefore take longer than the recorded query time.
Memory is Linux kernel peak RSS for the query worker through materialization, including initialization. It excludes the controller, reference checker, operating-system file cache and GPU VRAM. This avoids attributing a previous engine's retained allocations to the next engine.
Read the ranges, then the medians
The following direct-file measurements are median query seconds / maximum query-worker peak RSS in MiB, across five measured repetitions.
| Workload | Polars 2.0 default | DuckDB 2.0 | chDB 3.7.2 | pandas 2.2.2 |
|---|---|---|---|---|
| Original | 0.706 / 1,272 | 0.543 / 539 | 1.534 / 1,424 | 8.828 / 1,613 |
| Selective | 0.076 / 198 | 0.083 / 269 | 0.986 / 2,104 | 4.919 / 1,153 |
| High cardinality | 4.055 / 4,868 | 2.767 / 2,779 | 6.346 / 5,642 | 25.918 / 2,300 |
| Join | 0.936 / 1,046 | 0.349 / 283 | 1.671 / 1,405 | 15.909 / 2,324 |
For the original aggregation, DuckDB 2.0's observed query range was 0.489–0.735 seconds and default Polars 2.0's was 0.560–1.049 seconds. Those ranges overlap, so the run does not establish a speed winner between them. Their selective and high-cardinality ranges also overlap.
The join ranges were disjoint: DuckDB 2.0 took 0.243–0.394 seconds and default Polars 2.0 took 0.824–1.411 seconds. That is evidence about these implementations and five repetitions on this host. It is not a statistical significance test or a general ranking of the engines.
Memory changes the decision too. Default Polars used less peak RSS than DuckDB for the selective workload, while DuckDB used less for the other three cases. Pandas used less peak RSS than default Polars on the high-cardinality workload despite taking longer. A single speed or memory winner would hide these tradeoffs.
The full precision summaries and pairwise comparisons retain every configuration and observed range. Table memory values are rounded to whole MiB.
Version upgrades and execution modes
The harness includes Polars 1.8.2, Polars 2.0.0 with its default and explicit in-memory modes, and DuckDB 1.5.6 and 2.0.0. chDB, pandas, PyArrow and NumPy versions remain fixed across the two benchmark environments.
Default Polars 2.0 reduced both runtime and peak RSS compared with the legacy default for the original and high-cardinality workloads, with disjoint observed timing ranges. Its default execution mode also changed to streaming, so these observations do not isolate one optimizer improvement. The explicit in-memory control makes that distinction visible: the original workload used 4,552 MiB maximum RSS in-memory versus 1,272 MiB under the default.
DuckDB 1.5.6 and 2.0 have overlapping direct-file timing ranges in all four workloads. This run does not establish a direct-file speedup from that upgrade. Version features, compatibility and memory behaviour still matter alongside timing.
Persistent storage has an ingestion cost
DuckDB persistent tables and chDB MergeTree are measured in separate phases. Each trial bulk-loads a fresh store, closes it, and reopens it in a different query worker. Ingestion includes checkpointing and close. The one-shot cost combines ingestion, reopening and query execution.
For the selective workload, chDB's median direct-file query took 0.986 seconds. Its MergeTree query median was 0.421 seconds, but ingestion took 0.845 seconds and the one-shot median was 1.315 seconds. Repeated queries may amortize that cost; a single query does not automatically benefit.
Ingestion and query memory peaks also belong to different processes and must not be added. The high-cardinality MergeTree case peaked at 7,559 MiB during ingestion and 1,801 MiB during querying.
The persistent experiments describe the checked layouts, not a tuned ClickHouse server. The pandas baseline likewise does not claim to exhaust reader predicates or dtype tuning. These boundaries belong beside the results rather than in a footnote after a ranking.
Reproduce and inspect the evidence
The repository contains hash-locked legacy and modern environments, the workload definitions, process controller, independent oracle and report generator. After setting up those environments and downloading the inputs, the measured run uses:
.venv-bench-modern/bin/python -m benchmarks run \
--input 'dataset/*.parquet' --output-dir results/my-comparison
.venv-bench-modern/bin/python -m benchmarks report \
--output-dir results/my-comparison
The published manifest records input hashes, exact dependencies, hardware, settings, protocol and source revision. Raw trials retain warmups, measurements and failures. The portfolio bundles the checked summaries, manifest and pairwise comparisons; it does not download the full event dataset or execute these benchmarks in a visitor's browser.
The recorded run used source revision 18460f74d5a15db3081e6ce6b84ad57a2ae55c22; the evidence snapshot is pinned at packaging revision ab15af2fe034bd3b0f91f4d437eb606d786e0703. GPU execution, cold storage, remote input and data exceeding RAM need separate runs before I can make claims about them.