Skip to content

fix: account for NaN in float stats exported to DuckDB and DataFusion - #10479

Open
jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:fix/nan-aware-float-stats
Open

jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:fix/nan-aware-float-stats

Conversation

@jackylee-ch

@jackylee-ch jackylee-ch commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

What

Vortex's min, max and sum statistics skip NaN, but the DuckDB and DataFusion integrations report
them to the engine as exact bounds over every value. On a float column holding NaN, both engines
then answer from the statistics:

  • SELECT max(x) returns the largest non-NaN value instead of NaN;
  • WHERE x = 'NaN' returns no rows (DuckDB plans an EMPTY_RESULT, DataFusion prunes the file);
  • in DataFusion, where a negative NaN sorts below every value, WHERE x < 0.5 drops it as well.

Fix

Check the footer's exact NaNCount. DuckDB sorts NaN above every value, so its max becomes NaN
when the count is non-zero and is left out when the count is unknown; min is already correct.
DataFusion sorts NaN by sign, so min, max and sum are reported only for a NaN-free column (this
also guards the v2 source, which reports no statistics yet). Other columns are unchanged; float
columns in files from before NaNCount (0.29 to 0.32) lose their bounds.

The new NaN cases fail on develop (max(x) returns 2, the NaN filters return no rows); two plan
checks pin that NaN-free float columns are still answered from statistics.

constant_distinct_count (from #10448) now reuses is_nan_free. #10265 and #9775 rewrite these
functions.

AI assistance

Written with agentic AI assistance; each failure above was reproduced on develop before the fix.

Vortex's min, max and sum statistics skip NaN. Report them to DataFusion
only for a column whose footer NaNCount is exactly zero, and give DuckDB a
NaN max when the column holds NaN.

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005
robert3005 force-pushed the fix/nan-aware-float-stats branch from da90c5b to f2b1771 Compare October 15, 2026 17:36
@robert3005 robert3005 added the changelog/fix A bug fix label Oct 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants