feat(layout): execute a plan into a stream, add a Query plan, and benchmark against V1

[joseph-isaacs]

Summary

Part of the exec plan stack extracted from #10349. Stacked on #10418.

This PR finishes the node-based executor and measures it against the V1 scan (ScanBuilder/LayoutReader) like for like. It also adds the design document for the pipeline executor that the next PR implements.

Changes

• Streaming. execute runs a plan over a row range and yields arrays in row order.
• Query plan. It prunes zones, evaluates the conjuncts one at a time under the rows the earlier ones kept, then projects. Conjunct results are folded to bits as they arrive. The dense threshold is configurable.
• Planning. The optimizer memoizes shared subtrees. Arrays pass through nodes without copying. A Take keeps its values across executions.
• Decode cache. A cache bounded by generation lets segments that span splits be decoded once.
• Benchmarks. vortex-file/benches/query_exec.rs and query_v1.rs run on a lineitem-like file in several block-size and on-disk variants, split by layout or by fixed row ranges. vortex-file/examples/query_profile.rs runs interleaved A/B comparisons and counts allocations and reads.
• Design. vortex-layout/docs/pipeline-exec-design.md describes the push-based pipeline executor.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NWYmfHTFtaEdhdyd3u5hv3

---

Generated by Claude Code

[codspeed]

Merging this PR will degrade performance by 35.12%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 117 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

• exec[0, dict@plain-disk@64k]
• exec[0, dict@plain-disk@layout]
• exec[0, dict@zoned-disk@64k]
• exec[0, dict@zoned-disk@layout]
• exec[0, dict@zoned@layout]
• exec[0, project@plain-disk@64k]
• exec[0, project@plain-disk@layout]
• exec[0, project@zoned-disk@64k]
• exec[0, project@zoned-disk@layout]
• exec[0, q6_flat@plain-disk@64k]
• …and 107 more

❌ 1 regressed benchmark
✅ 2152 untouched benchmarks
🆕 310 new benchmarks
⏩ 381 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	decode_primitives[i64, (1000, 8)]	24.6 µs	37.9 µs	-35.12%
🆕	Simulation	exec[0, dict@plain-256@64k]	N/A	1.3 s	N/A
🆕	Simulation	exec[0, dict@plain-256@layout]	N/A	1.6 s	N/A
🆕	Simulation	exec[0, dict@plain-2k@64k]	N/A	473 ms	N/A
🆕	Simulation	exec[0, dict@plain-2k@layout]	N/A	512.4 ms	N/A
🆕 ⚠️	Simulation	exec[0, dict@plain-disk@64k]	N/A	376.7 ms	N/A
🆕 ⚠️	Simulation	exec[0, dict@plain-disk@layout]	N/A	397.6 ms	N/A
🆕	Simulation	exec[0, dict@plain@64k]	N/A	370.8 ms	N/A
🆕	Simulation	exec[0, dict@plain@layout]	N/A	387.9 ms	N/A
🆕 ⚠️	Simulation	exec[0, dict@zoned-disk@64k]	N/A	376.9 ms	N/A
🆕 ⚠️	Simulation	exec[0, dict@zoned-disk@layout]	N/A	399.9 ms	N/A
🆕	Simulation	exec[0, dict@zoned@64k]	N/A	371 ms	N/A
🆕 ⚠️	Simulation	exec[0, dict@zoned@layout]	N/A	390.3 ms	N/A
🆕	Simulation	exec[0, project@plain-256@64k]	N/A	517.1 ms	N/A
🆕	Simulation	exec[0, project@plain-256@layout]	N/A	520.6 ms	N/A
🆕	Simulation	exec[0, project@plain-2k@64k]	N/A	162.9 ms	N/A
🆕	Simulation	exec[0, project@plain-2k@layout]	N/A	165.3 ms	N/A
🆕 ⚠️	Simulation	exec[0, project@plain-disk@64k]	N/A	115.7 ms	N/A
🆕 ⚠️	Simulation	exec[0, project@plain-disk@layout]	N/A	115.7 ms	N/A
🆕	Simulation	exec[0, project@plain@64k]	N/A	113.7 ms	N/A
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ji/exec-6-execute-stream (6eb349c) with ji/exec-5-list-pack-node (152edd4)}

Footnotes

1. 381 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩