Repository navigation
fix: omit zone-map stats that a nullable struct's nulls would corrupt - #10413
Conversation
Struct fields are written unmasked: a field still holds a value in rows where the containing struct is null, and the struct validity is written as a separate child. Each field's zone map was computed over that unmasked stream, so null_count undercounted the logical column (it was always 0 for a non-nullable field under a nullable struct) and nan_count counted values the column never shows. Pruning on `s.x IS NULL` then discarded zones that held matching rows. Add `LayoutWriterContext::with_nullable_ancestor`, set by `StructStrategy` on the context it hands to field writers of a nullable struct, and have `ZonedStrategy` keep only min/max style aggregates under it. Those remain sound as bounds because the unmasked field is a superset of the logical values. Count aggregates are omitted rather than recorded wrong until they can be computed over the masked values. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HRQvL72sLkXczGCjvFWaMG
Pin down that the flag set by a nullable struct reaches leaves below any number of non-nullable struct levels, is not set above the nullable level, and leaves sibling leaves untouched. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HRQvL72sLkXczGCjvFWaMG
Merging this PR will regress 1 benchmark
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | bitpack_blocked_compress_avx512 |
4.9 µs | 5.7 µs | -13.94% |
| ⚡ | WallTime | threshold_neon[8] |
4 µs | 3 µs | +33.84% |
| ⚡ | WallTime | filter_neon[64] |
3.5 µs | 2.9 µs | +19.76% |
| ⚡ | WallTime | threshold_neon[80] |
5.8 µs | 4.8 µs | +19.15% |
| ⚡ | WallTime | threshold_neon[64] |
5.3 µs | 4.5 µs | +18.52% |
| ⚡ | WallTime | filter_neon[80] |
3.5 µs | 3.2 µs | +11.25% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ji/magical-bell-b84xbc (d00d4e2) with develop (9d5a844)
Zone maps hold exact values. Min and max over an unmasked field are sound bounds for the logical column but can be wider than its true range, because rows the struct nulls out still hold a value. Rather than keep them as inexact bounds, skip the zone map entirely for such fields and write the data child alone, until the aggregates are computed over the masked values. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HRQvL72sLkXczGCjvFWaMG
The context carries tree-position facts such as permitted aggregates and the nullable-ancestor flag, which reach a child only because its parent forwarded them. Say so on the trait so a strategy that builds a fresh context is caught in review. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HRQvL72sLkXczGCjvFWaMG
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HRQvL72sLkXczGCjvFWaMG
Struct fields are written unmasked: a field still holds a value in rows where the containing struct is null, and the struct validity is written as a separate child. Each field's zone map was computed over that unmasked stream, so null_count undercounted the logical column (it was always 0 for a non-nullable field under a nullable struct) and nan_count counted values the column never shows. Pruning on
s.x IS NULLthen discarded zones that held matching rows.Add
LayoutWriterContext::with_nullable_ancestor, set byStructStrategyon the context it hands to field writers of a nullable struct, and haveZonedStrategykeep only min/max style aggregates under it. Those remain sound as bounds because the unmasked field is a superset of the logical values. Count aggregates are omitted rather than recorded wrong until they can be computed over the masked values.Claude-Session: https://claude.ai/code/session_01HRQvL72sLkXczGCjvFWaMG
Summary
Changes