[joseph-isaacs]

Summary

DuckDB provides runtime join filters, optional-filter sampling policies, and scan-order hints that the Vortex integration did not fully consume. This change uses those inputs, improves compressed predicates and hash-build selection, and memory-maps immutable local files to avoid positional reads and compressed-buffer copies.

Updated onto develop c1a2ca1262, including develop's SQL IN/NOT IN null semantics (#10054).

Performance

Current local x86_64 results at 9e1c1fe4: all 113 JOB queries, eight DuckDB threads, five iterations, all three formats stored as persistent local files. Hot time is the sum of each query's median over iterations 2–5; these are warm-cache measurements.

Format	Hot time	Format / Vortex	Vortex wins
Vortex	6.454 s	1.00×	—
Parquet	25.543 s	3.96×	113 / 113
DuckDB native	10.433 s	1.62×	88 / 113

The isolated local reader experiment before the final develop merge reduced Vortex hot time from 8.499 s to 6.423 s (24.4% lower), with all 113 query medians improved. An ABBA recheck of six targeted queries with 11 iterations per invocation reduced their combined hot time by 15.9%.

Cold-file reader comparisons on 7b, 16a, and 25b reduced Vortex time by 37.5%, 9.4%, and 12.7%. Each process started after per-file cache eviction; mincore verified zero resident pages across all 21 Vortex inputs before launch. These compare Vortex's positional and mapped readers, and do not compare cold Parquet or native DuckDB. Binding/metadata setup remains outside the query timer.

Earlier GitHub JOB on NVMe results at 6a7cb232 used ARM hardware and precede the memory-map change. Vortex hot time was 4.753 s versus 26.130 s for Parquet and 6.920 s for native. Compared with PR #10484's matching ARM run, Vortex improved from 5.387 s to 4.753 s (11.8%). That revision comparison includes intervening develop changes. Absolute ARM and local x86 times are separate comparisons.

Changes

• Evaluate owned snapshots of finalized DuckDB Bloom filters with native key hashing, packed result bits, reusable scratch vectors, AVX-512 dispatch where available, and scalar/nullable fallbacks.
• Apply DuckDB's optional-filter thresholds and row budgets. Preserve supported siblings of unsupported optional conjunctions and consumed build-side predicate hints.
• Honor scan-order hints, null ordering, and legacy column statistics. Preserve split/row coordinates through ordering and pruning.
• Prefer develop's prepared-set and SQL-null membership implementation (Optimize list_contains with prepared constant sets #10052, Use SQL null semantics for DuckDB IN and NOT IN pushdown #10054), retain the dictionary/run-end follow-up (perf: push a prepared list_contains set into Dict and RunEnd needles #10442), and probe small sets directly in FSST/OnPair compressed values. Add ordered multi-literal LIKE matching over FSST.
• Restore native runtime-filter targets through exactly representable small-integer-to-DOUBLE equality casts, including signed-zero normalization; exclude inexact wide casts and null-safe joins.
• Prefer smaller variable-payload hash builds in supported Vortex inner-equality plans, with guards for compact builds and inflated estimates. Rebuild projection lifetimes and runtime-filter metadata after swaps.
• Memory-map immutable local files while retaining the shared footer cache and mapping ownership through exported arrays. Document the immutable-input requirement and VORTEX_DUCKDB_MEMORY_MAP=0 positional-read fallback. Remote scans retain their existing reader.
• Restrict exact distinct statistics to proven unique integer columns, apply the pushed-filter cardinality discount once, and keep profiling builds optimized.

Validation

• 429 DuckDB tests passed; two skipped, including upstream SQL-null membership cases and new mapped-buffer lifetime/reopen/path tests.
• Actual results match across Vortex, Parquet, and native DuckDB for all 113 JOB queries after the develop merge.
• cargo +nightly fmt --all, cargo clippy --all-targets --all-features -- -D warnings, and DuckDB doctests passed.
• Prior ClickBench verification passed all 43 queries against unchanged result snapshots in both formats, full-row/schema parity, and four cardinality regression cases.

25 JOB queries still lose to native in the latest local run. All measurements use DuckDB 1.5.5. The newer development-line prefix-range runtime join filter is not yet integrated; that requires an updated DuckDB bridge.