Repository navigation
Conversation
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Merging this PR will improve performance by 26.27%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ |
Simulation | density_sweep_dense_runs[0.001] |
47.4 µs | 29 µs | +63.38% |
| ⚡ | WallTime | filter_neon[8] |
1,216 ns | 880 ns | +38.18% |
| ⚡ | WallTime | filter_neon[64] |
2.3 µs | 1.7 µs | +32.29% |
| ⚡ | WallTime | threshold_neon[8] |
1.8 µs | 1.5 µs | +25.6% |
| ⚡ | WallTime | threshold_neon[80] |
5.4 µs | 4.3 µs | +23.25% |
| ⚡ ℹ️ | Simulation | test_scan_filter[int] |
3.7 ms | 3.1 ms | +19.59% |
| ⚡ |
Simulation | random_i128[0.01] |
5.9 µs | 5.1 µs | +17.66% |
| ⚡ | WallTime | bitpack_blocked_compress_avx2 |
7.6 µs | 6.7 µs | +13.34% |
| ⚡ | WallTime | threshold_neon[64] |
2.9 µs | 2.6 µs | +10.68% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing mk/list-layout-split-rows (a488121) with mk/list-layout-edition-gate (a1bb97f)
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.868x ✅, 1↑ 0↓)
File Size Changes (1 files changed, -0.6% overall, 0↑ 1↓)
Totals:
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.972x ➖, 4↑ 2↓)
duckdb / parquet / ns (0.989x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 1↑ 3↓)
duckdb / parquet / ns (0.976x ➖, 3↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.006x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.996x ➖, 4↑ 1↓)
duckdb / parquet / ns (0.994x ➖, 2↑ 2↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.096x ➖, 0↑ 6↓)
datafusion / parquet / ns (1.076x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.024x ➖, 1↑ 3↓)
duckdb / parquet / ns (1.076x ➖, 0↑ 0↓)
|
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.869x ✅, 1↑ 0↓)
File Size Changes (1 files changed, -0.6% overall, 0↑ 1↓)
Totals:
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.024x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (1.076x ➖, 0↑ 3↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.939x ➖, 3↑ 2↓)
duckdb / vortex-compact / ns (0.953x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.990x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.000x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (0.993x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.015x ➖, 1↑ 2↓)
No file size changes detected. |
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.896x ➖, 3↑ 1↓)
datafusion / vortex-compact / ns (1.017x ➖, 0↑ 2↓)
datafusion / parquet / ns (1.316x ❌, 0↑ 5↓)
duckdb / vortex-file-compressed / ns (1.041x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (1.005x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.026x ➖, 0↑ 1↓)
|
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.988x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.012x ➖, 1↑ 1↓)
duckdb / duckdb / ns (1.001x ➖, 0↑ 0↓)
File Size Changes (201 files changed, +0.0% overall, 107↑ 94↓)
Totals:
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.979x ➖, 4↑ 0↓)
duckdb / parquet / ns (0.977x ➖, 4↑ 0↓)
duckdb / duckdb / ns (0.997x ➖, 0↑ 1↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.009x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.007x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.989x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.011x ➖, 0↑ 1↓)
duckdb / duckdb / ns (1.003x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Benchmarks: JOB on NVME 📖Commits: PR No baseline is available for this benchmark yet; PR measurements are shown without comparison. duckdb / vortex-file-compressed / ns (no group data, 0↑ 0↓)
duckdb / parquet / ns (no group data, 0↑ 0↓)
duckdb / duckdb / ns (no group data, 0↑ 0↓)
No baseline file sizes found for base commit. |
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.398x ✅, 10↑ 0↓)
datafusion / vortex-compact / ns (0.328x ✅, 10↑ 0↓)
datafusion / parquet / ns (1.009x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.430x ✅, 9↑ 0↓)
duckdb / vortex-compact / ns (0.400x ✅, 9↑ 0↓)
duckdb / parquet / ns (1.017x ➖, 0↑ 0↓)
File Size Changes (2 files changed, -7.1% overall, 0↑ 2↓)
Totals:
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.996x ➖, 1↑ 0↓)
duckdb / vortex-compact / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.012x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.000x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.994x ➖, 5↑ 1↓)
duckdb / vortex-compact / ns (1.003x ➖, 2↑ 6↓)
duckdb / parquet / ns (1.000x ➖, 4↑ 4↓)
duckdb / duckdb / ns (0.996x ➖, 5↑ 1↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.889x ➖, 3↑ 1↓)
datafusion / vortex-compact / ns (0.900x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.959x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (0.921x ➖, 2↑ 1↓)
duckdb / vortex-compact / ns (0.786x ➖, 4↑ 0↓)
duckdb / parquet / ns (0.993x ➖, 0↑ 0↓)
|
Benchmarks: Vortex queries 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.979x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.826x ➖, 6↑ 1↓)
datafusion / vortex-compact / ns (0.842x ➖, 5↑ 2↓)
datafusion / parquet / ns (1.035x ➖, 1↑ 5↓)
duckdb / vortex-file-compressed / ns (0.784x ➖, 4↑ 0↓)
duckdb / vortex-compact / ns (0.907x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.803x ➖, 5↑ 0↓)
|
Summary
Stacked on #10529.
ListReaderplaces scan splits on a grid of outer rows, and the grid was fixed at 64 cells whatever the file size. The grid now grows with the row count: at least 64 cells, and enough that a cell covers at most about 8,192 rows. Files under about 512K rows are unchanged.How list splits work
register_splitsmust report split points in outer rows, but a list's natural boundaries are its element blocks, which are cut in element rows. It can't read the offsets at that point, so it estimates:Example: 800 outer rows, 4,000 elements in 5 blocks, an 8-cell grid:
Grid size (this PR):
cells = max(64, next_power_of_two(ceil(rows / 8192)))Results
Local runs, list layout ÷ develop (52a1d15): 3 interleaved rounds with rotating order, in-process iterations, Apple M5 Max (18 cores), warm page cache,
release_debugwithtarget-cpu=native. "grid 64" is the same list layout without this PR's reader change.vortexvortexvortexvortex-compactvortex-compactpolarsignals, DataFusion, `vortex` (20 iterations)
statpopgen, DataFusion, `vortex` (5 iterations)
statpopgen, DuckDB, `vortex` (5 iterations)
statpopgen, DataFusion, `vortex-compact` (5 iterations)
statpopgen, DuckDB, `vortex-compact` (5 iterations)
File sizes (list layout ÷ develop): statpopgen
vortex1.93 → 1.74 GB (0.90x), statpopgenvortex-compact1.003 → 0.977 GB (0.97x), polarsignals 718.8 → 714.2 MB (0.99x).