[myrrc]

ScalarFn validity() recursively canonicalizes whole subgraph starting from
itself. This is very inefficient. For example, if you have a single probe of
element at index i, the whole array will be canonicalized. What's worse, probing
and scalar_at check element's validity before getting the element, and for some
functions like list_contains this evaluates the function[1], so for every probe you
end up evaluating the function twice.

We already have ScalarFn reduce<ReduceNode> which allows you to write unified
rules over both expressions and arrays. This change makes validity also
parametrised by a ReduceNode parameter and also adds a validity() method on the
ReduceNode itself.

There are two ways how you can reduce a node. First is symbolic reduction which
happens without executing anything - think replacing IsNotNull(x) with
x.validity. If you can't reduce the node anymore, you do one step of evaluation -
think calling execute() - and then repeat. This is how our executor works in
general, and same logic may be applied to ReduceNodes. validity<T> returns
these two cases: a node can either be symbolically reduced to other node, or is
Irreducible in which case you need to execute.

While probing we then avoid second function evaluation - if expression/array's
validity() is Irreducible, evaluating function once is sufficient.

We also want the execution of validity() to be lazy - it should never fully
canonicalize. Unfortunately, ScalarFn::execute() ultimately returns "done" so
you can't express "give me my child and execute it one step". Thus this single
step execution happens inside an internal Array which is exactly "validity of an
Irreducible ScalarFn array".

Expression's validity() for a scalar function returns either a reduced
expression or IsNotNull(self). In the latter case we want to avoid infinite
recursion when replacing IsNotNull(x) -> x.validity() so the reduction rule
does nothing if the expression returned Irreducible.

To avoid manually rewriting all existing validity rules, we can use the
predicate "if a function is strict[2] and all nullable argument's return types
are non-nullable, then its validity is and() over child validities". This
removes a significant number of overloads.

This change also adds a is_null/is_not_null reduce rule which is
#10025 but reimplemented
on top of generic reduce<> which requires validity<>.

Given X = bool?, Y = bool, it symbolically reduces

is_null(and(X, Y)) -> and(is_null(X), Y)
is_not_null(and(X, Y)) -> or(X.validitY, not(Y))
is_null(or(X, Y)) -> and(is_null(X), not(Y))
is_not_null(or(X, Y)) -> or(X.validitY, Y)

during array.optimize(). These optimizations lower the complexity of subtree
evaluation. As an example, is_null(and(x, y)) is 5 computations: x, y, and(x,
y), and(x, y).validity(), not(and(x, y).validity()). We replace it with 4:
x.validity, y, not(x.validity), and(not(x.validity()), y). Given x.validity is
either as expensive or less expensive than x, this is a net benefit.

This change also doesn't execute the short-circuited array in fill_null. As
scalar fn validity is now lazy, we can safely return the lazy array with added
cast.

This change also adds the literal stats falsifier. As scalar function evaluation
may now fold into a constant, we need to falsify the partition based on this
literal.

[1] list_contains([], null) = false, #10028, but to know this, you need to
evaluate x's offsets.
[2] zip is a special case because it satisfies the return type condition but
isn't strict.

[codspeed]

Merging this PR will degrade performance by 3.06%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 3 improved benchmarks
❌ 5 regressed benchmarks
✅ 2054 untouched benchmarks
🆕 8 new benchmarks
⏩ 503 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	WallTime	mul_u64_nonnull_neon	29.6 µs	39.4 µs	-24.99%
❌	Simulation	execute_scalar_struct_wide	403.5 µs	470.2 µs	-14.19%
❌	WallTime	multiply_shapes_neon[(32768, PerRowPerRow)]	33.3 µs	38.5 µs	-13.74%
❌	WallTime	mul_i64_nonnull_neon	33.5 µs	38.5 µs	-13.03%
❌	Simulation	execute_scalar_struct_simple	93.4 µs	106.5 µs	-12.31%
⚡	Simulation	bench_dict_mask[(0.9, 0.9)]	771.7 µs	516.1 µs	+49.53%
⚡	WallTime	dict_canonicalize_gt_u8_neon[1000000]	547 µs	488.8 µs	+11.91%
⚡	WallTime	scalar_subtract_neon	13.9 µs	12.6 µs	+10.03%
🆕	Simulation	probe_scalar_fn_once[vortex.binary(bool?, len=100000)]	N/A	303.3 µs	N/A
🆕	Simulation	probe_scalar_fn_once[vortex.binary(i64?, len=100000)]	N/A	720.1 µs	N/A
🆕	Simulation	probe_scalar_fn_repeated[vortex.binary(bool?, len=100000)]	N/A	315.8 µs	N/A
🆕	Simulation	probe_scalar_fn_repeated[vortex.binary(i64?, len=100000)]	N/A	728.5 µs	N/A
🆕	Simulation	probe_scalar_fn_valid_once[vortex.binary(bool?, len=100000)]	N/A	372.6 µs	N/A
🆕	Simulation	probe_scalar_fn_valid_once[vortex.binary(i64?, len=100000)]	N/A	411.8 µs	N/A
🆕	Simulation	probe_scalar_fn_valid_repeated[vortex.binary(bool?, len=100000)]	N/A	216.6 µs	N/A
🆕	Simulation	probe_scalar_fn_valid_repeated[vortex.binary(i64?, len=100000)]	N/A	67 µs	N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing myrrc/lazy-scalarfn-validity (2b9f8c6) with develop (a96e0a8)}

Footnotes

1. 503 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[myrrc]

In addition to the discussion: there are three places where you can do the single execute step once all symbolic reduction rules are done: 1. Executor pipeline. We can't do this right now since ScalarFn::execute returns "done", and teaching it to do ExecuteSlot's is a separate big change. 3. IsNotNull::validity() (chosen). ScalarFn::validity() stays lazy, but IsNotNull::validity() may do execution, and you need scalar special handing in order to avoid infinite recursion. 2. ScalarFn::validity(). No scalar special handling, but validity() isn't fully lazy, as it may do execution

[myrrc]

the flaw of current API is that repeated probe cache is same for both masks and values, so you can't just cache the validity.

We've agreed I'd fix this separately. There are two separate changes to remove both checks for .is::<ScalarFn>(). 1. RepeatedProbe should be able to ask the array implementing OperationsVTable for validity, and the array should be responsible for providing the validity, canonicalized or lazy. 2. probe_scalar requires the caller has checked the validity. We can remove this precondition and instead make probe_scalar check the validity itself when it wants.

[myrrc]

Regarding IsNotNull special handling: there is an issue indeed if you add a new object, say, WindowFunction, and this WindowFunction has special handling for is_not_null, there will be an issue with cycles again, and we'd need to handle this in IsNotNull execution, which is bad. So, as a temporary measure, we'll merge a solution where Array.validity() may do an execute step, and then apply an optimization on top of it: a special internal array which is constructed when ScalarFn.validity() is Irreducible. On execution of this special array it replaces itself with `is_not_null(x.execute::<ArrayRef>())`.

[joseph-isaacs]

could avoid the unwrap and alloc

[joseph-isaacs]

hopefully we avoid an alloc for a lit bool