concurrent_qps measures swap-thrashing, not throughput, for single-process engines (introduced in #860)

[george-larionov]

Summary

The concurrent_qps test introduced in #860 produces misleading, non-reproducible numbers for single-process / in-process engines (embedded CLIs and dataframe-server systems). For these systems it spawns N independent full-machine processes that collectively oversubscribe RAM, so on memory-constrained instances the metric measures swap-thrashing throughput, not query throughput — and whether it completes at all depends on the runner environment rather than the engine.

This surfaced in #943 (DataFusion), but the root cause is in lib/benchmark-common.sh (bench_concurrent_qps) and affects a whole class of systems.

Evidence

bench_concurrent_qps launches BENCH_CONCURRENT_CONNECTIONS (default 10) workers for BENCH_CONCURRENT_DURATION (default 600s). For BENCH_DURABLE=no engines like datafusion-cli, each worker's ./query spawns a fresh process that assumes it owns the whole machine. 10 × ~7 GB RSS ≈ 70 GB against 32 GB RAM.

Checked-in DataFusion (single) results, concurrent_qps vs. machine RAM:

Machine	vCPU / RAM	concurrent_qps	err
c6a.xlarge	4 / 8 GB	0.025	0
c6a.2xlarge	8 / 16 GB	0.042	0
c6a.4xlarge	16 / 32 GB	0.11	0
c8g.4xlarge	16 / 32 GB	0.092	0
c6a.metal	192 / 384 GB	4.017	0
c8g.metal-48xl	192 / 384 GB	5.388	0
c7a.metal-48xl	192 / 384 GB	5.888	0

(datafusion-partitioned shows the same shape: 0.025–0.097 on the small boxes, 4.9–7.0 on the metals.)

The QPS tracks RAM, not core count — the metric on the sub-RAM machines is reporting how fast the box can page, not how fast the engine runs queries. err=0 everywhere, so nothing crashed in the automated runs; the processes just crawled through swap.

Why the same instance type gives different outcomes

cloud-init.sh.in adds a 16 GB swapfile and installs earlyoom on launch. In the automated fleet run a c6a.4xlarge therefore thrashes through swap and finishes at ~0.11 qps. A contributor running ./benchmark.sh on a bare c6a.4xlarge (no swap) gets the kernel OOM-killing datafusion-cli outright (see #943). So a published number depends on the presence of swap in the runner environment — a reproducibility problem independent of any one engine.

Scope

The same dynamic applies to every BENCH_DURABLE=no single-process / in-process system where N independent processes oversubscribe RAM — e.g. duckdb (and dataframe/memory variants), sqlite, hyper, chdb, pandas, polars-dataframe, daft, sirius. Any of these on a sub-RAM machine will show the same artifact.

Options

1. Run the concurrent test at concurrency = 1 for single-process engines. Measures real serial throughput, stays comparable, never oversubscribes RAM. (Suggested by @alamb in fix(datafusion): Update docs, skip concurrent_qps, Skip re-downloading partitioned files #943.)
2. Disable concurrent_qps for the whole single-process / in-process class. fix(datafusion): Update docs, skip concurrent_qps, Skip re-downloading partitioned files #943 does this for DataFusion only (BENCH_CONCURRENT_DURATION=0); make it class-wide so the dashboard isn't uneven (one embedded engine reporting null while peers report swap artifacts).
3. Add a "fit in RAM" guard analogous to the partial-load >= 5 GB check: if the concurrent test's working set exceeds available RAM, record null instead of a thrashing number.
4. Pin the environment: document/encode whether swap is part of the canonical setup so the same instance type reproduces the same result. (Weakest on its own — a number that only completes because of swap shouldn't be published regardless.)

Recommendation: option 1 or 2 for the single-process class (subsumes #943 into a consistent, class-wide fix), plus nulling the existing swap-artifact concurrent_qps values already checked in for these systems on the RAM-constrained machines.

Related

• Refactor: standard install/start/check/stop/load/query interface per system #860 — introduced the concurrent QPS test
• fix(datafusion): Update docs, skip concurrent_qps, Skip re-downloading partitioned files #943 — DataFusion-specific workaround that prompted this

[alexey-milovidov]

The concurrent_qps is a supplementary metric, it is not used in the reports.
It is okay if it has complications with non-concurrent systems. At least it is better than complicating the scripts.

[alexey-milovidov]

Though the benchmark script should at least succeed...

[alexey-milovidov]

Let's do this: add a variable to skip the concurrency test.