Repository navigation
fix: account for NaN in float stats exported to DuckDB and DataFusion - #10479
Open
jackylee-ch wants to merge 1 commit into
Open
jackylee-ch wants to merge 1 commit into
jackylee-ch wants to merge 1 commit into
Conversation
Vortex's min, max and sum statistics skip NaN. Report them to DataFusion only for a column whose footer NaNCount is exactly zero, and give DuckDB a NaN max when the column holds NaN. Signed-off-by: jackylee-ch <qcsd2011@gmail.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005
force-pushed
the
fix/nan-aware-float-stats
branch
from
October 15, 2026 17:36
da90c5b to
f2b1771
Compare
robert3005
approved these changes
Oct 16, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Vortex's min, max and sum statistics skip NaN, but the DuckDB and DataFusion integrations report
them to the engine as exact bounds over every value. On a float column holding NaN, both engines
then answer from the statistics:
SELECT max(x)returns the largest non-NaN value instead of NaN;WHERE x = 'NaN'returns no rows (DuckDB plans anEMPTY_RESULT, DataFusion prunes the file);WHERE x < 0.5drops it as well.Fix
Check the footer's exact
NaNCount. DuckDB sorts NaN above every value, so its max becomes NaNwhen the count is non-zero and is left out when the count is unknown; min is already correct.
DataFusion sorts NaN by sign, so min, max and sum are reported only for a NaN-free column (this
also guards the v2 source, which reports no statistics yet). Other columns are unchanged; float
columns in files from before
NaNCount(0.29 to 0.32) lose their bounds.The new NaN cases fail on
develop(max(x)returns 2, the NaN filters return no rows); two planchecks pin that NaN-free float columns are still answered from statistics.
constant_distinct_count(from #10448) now reusesis_nan_free. #10265 and #9775 rewrite thesefunctions.
AI assistance
Written with agentic AI assistance; each failure above was reproduced on
developbefore the fix.