[robert3005]

Stacked on #10052.

What

The set that list_contains looks needles up in is now an array. ArrayHaystack holds a lookup array: an array whose encoding is built to find needles and which canonicalizes to the set's elements. Before this change it held a private Box<dyn HaystackState>.

This answers the review question on #10052, "how do I extend this to work for new custom kernels?":

• Bring your own lookup array. Any encoding that implements the public HaystackVTable (contains(array, needles, ctx) -> (BitBuffer, Validity)) can be wrapped with ArrayHaystack::new. PreparedSetData::try_with_haystack(list, haystack) prepares a set that uses it.
• Read the lookup array from a needle kernel. PreparedSetData::haystack(ctx)?.array() gives the lookup array, so a kernel can downcast to an encoding it knows.

Built-in lookup arrays

All four are crate-private and exist only in memory: serialize/deserialize fail, and they are not registered with ArraySession.

Encoding	Elements	Children / data
SortedHaystack	integers, floats by bit pattern, decimals by unscaled value	the values, sorted without duplicates
BitmapHaystack	the same, over a dense span	the smallest value (one row) and a BoolArray bitmap
BytesHaystack	UTF-8 / binary	the VarBinView elements; the hash tables are the array's in-memory data
RowsHaystack	anything else	recursively canonical elements and the u64 indices of the distinct elements in sorted order

ArrayHaystack::try_new picks the encoding with the same rules as before, and the lookup loops are unchanged. A set without its own haystack still builds one lazily on the first lookup, and it is shared across batches, chunks and slices. PreparedSetData now keeps its elements and its haystack in separate lazy cells.

Notes for review

• ArrayHaystack, HaystackVTable, PreparedSetData::try_with_haystack and PreparedSetData::haystack are new public API. The built-in encodings stay pub(crate).
• A lookup array has no null rows, and it may repeat an element (BytesHaystack does).
• Each lookup array reads a single row through its canonical form, since lookup arrays exist for lookups, not for reads.

Checks

• cargo +nightly-2026-09-10 fmt -p vortex-array
• cargo clippy -p vortex-array -p vortex-sequence --all-targets --all-features -- -D warnings: clean
• cargo nextest run -p vortex-array -p vortex-sequence: 4282 passed, 1 skipped (10 new tests in array/haystack/tests.rs, including a lookup array of a test-only encoding)
• RUSTDOCFLAGS="-D warnings" cargo doc -p vortex-array --no-deps: no new warnings (it fails only on the existing trace_macros.rs links)
• typos: clean
• Not run: the list_contains_set benchmarks, doctests, and workspace-wide checks.

🤖 Generated with Claude Code

[codspeed]

Merging this PR will regress 1 benchmark

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 1 benchmark measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. This result is not comparable, so it counts as unchanged.

Preventing compiler optimizations

• bench_compare_primitive[(10000, 2)]

⚠️ 36 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

• test_scan_scalar_at[int_str]
• random_i128[0.01]
• test_repeated_scan_filter[int]
• test_repeated_scan[int]
• test_repeated_scan_to_arrow_array[int]
• density_sweep_dense_runs[0.001]
• chunked_dict_primitive_canonical_into[f64, (1000, 1000, 10)]
• test_repeated_scan_scalar_at[int]
• test_scan_filter[int_str]
• test_scan_filter[int_float]
• …and 26 more

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 12 improved benchmarks
❌ 1 regressed benchmark
✅ 2246 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌ ⚠️	Simulation	test_scan_scalar_at[int_str]	1.2 ms	1.3 ms	-13.02%
⚡	WallTime	filter_neon[64]	3.4 µs	2.8 µs	+23.2%
⚡	WallTime	filter_neon[80]	3.6 µs	2.9 µs	+20.67%
⚡	WallTime	threshold_neon[80]	5.6 µs	4.7 µs	+18.75%
⚡	WallTime	threshold_neon[8]	3.5 µs	3 µs	+18.14%
⚡ ⚠️	Simulation	random_i128[0.01]	7.6 µs	6.5 µs	+17.34%
⚡	Simulation	test_scan_filter_to_arrow_array[int_float]	4.2 ms	3.6 ms	+15.22%
⚡	WallTime	bitpack_blocked_compress_avx2	7.7 µs	6.7 µs	+14.23%
⚡	WallTime	filter_avx512[64]	2.5 µs	2.2 µs	+12.48%
⚡	Simulation	test_repeated_scan_filter[int_float]	4 ms	3.5 ms	+12.03%
⚡	Simulation	test_scan_filter_to_arrow_array[int]	2.8 ms	2.5 ms	+10.96%
⚡	WallTime	threshold_neon[64]	4.8 µs	4.4 µs	+10.22%
⚡	Simulation	test_scan_filter[int]	3.1 ms	2.8 ms	+10%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing rk/haystack-arrays (20e06e2) with develop (c1a2ca1)}