You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix!: DH-23754: Reject lossy filter value coercion, read char integers as code points - #8726
BREAKING CHANGE: MatchFilter and RangeFilter no longer truncate, wrap or saturate a filter value to fit the column's type. Each change below replaces a result that disagreed with the query language, that is, with the ConditionFilter these filters fail over to.
Now throws instead of selecting the wrong rows. These forms have no failover:
X in v with a lossy query-scope value. With v = 5.7 on an int column, main matched 5.
A lossy direct value. new MatchFilter(REGULAR, "X", 300) on a byte column matched 44 on main.
A lossy Filter API in or eq literal, such as FilterIn.of(X, 4, 5.5).
Now selects different rows:
X == v, X != v and range filters with a lossy query-scope value fail over to the query language. So X < v with v = 5.7 on an int column selects 4 and 5, where main selected 4.
On a char column, an unquoted integer literal is a code point: X == 5 is (char) 5. Write X == '5' for the digit.
A char compared with a numeric column is its code point, as in Java. The Filter API's lt(X, '5') on an int column compares with 53, where main used 5.
Split out of DH-23754. Rejecting values of the wrong type (checkValueTypes) is left to a separate PR, because it reverses the "Will not implement" decision on DH-21232.
The rule
MatchFilter and RangeFilter converted a query-scope value to the column's type with the query language's casts, which truncate, wrap and saturate. With v = 5.7 on an int column, X == v matched 5. With v = 300 on a byte column, it matched 44. The same condition written as a formula matches nothing.
A value must now convert exactly, meaning the converted value selects exactly the rows the query language would. Otherwise it is refused:
X == v, X != v and range filters fail over to their ConditionFilter, which is always correct.
X in v, direct values and Filter API values have no failover, so they throw.
Changes
Numeric columns. A single NumericColumnTypeConvertor handles byte through double. It casts the value (narrow), then checkRoundTrip compares the result with the original exactly, as BigDecimals. It also refuses:
a value that is not a number, which fails over as it did on main, where the cast threw ClassCastException;
a Number of a nonstandard type, such as DoubleAdder or AtomicLong, which has no exact value to check.
Chars. The query language, like Java, compares a char with a number by its code point. On numeric columns, byte through double, BigInteger and BigDecimal:
A Character query-scope or direct value converts to its code point, and so does a quoted char literal such as '5'.
A code point too large for a byte or short column is refused.
NULL_CHAR is null.
On a char column, a number converts to the char with that code point, and NULL_CHAR (65535) is refused.
BigDecimal and BigInteger columns convert exactly. A floating-point value converts through BigDecimal.valueOf, as the query language compares it, and a long no longer goes through double. NaN converts to neither, so X == v fails over, where NaN equals no big number, and X in v throws. main matched 0 on a BigInteger column.
A BigDecimal or BigInteger on a float or double column. The query language compares these through BigDecimal.valueOf(double), which uses the column value's shortest decimal representation. So the value converts only if it equals that decimal for its converted value:
BigDecimal("0.1") converts to 0.1.
new BigDecimal(0.1) is refused. It is exactly a double, but no double's shortest decimal equals it.
BigDecimal("0.1") on a float column is refused, because 0.1f widens to 0.10000000149011612.
Direct values, such as new MatchFilter(options, "X", 5.7), now go through the same conversion. Before, they were used as given. Without nanMatch, NaN is now dropped from query-scope values on primitive columns, as it already was from direct values.
Range bounds.-0.0 against a byte, short, int or char column fails over. The query language compares these with Double.compare, which orders -0.0 below 0, so the converted bound 0 would select other rows. The RangeFilter javadoc documents this case, the remaining differences listed under "Not changed", and how null endpoints behave.
Values of another type. The convertors pass a value they do not convert through as it is, such as a String on a BigInteger column. It can never match, so MatchFilter now drops it from its values on every column type, not only BigDecimal. That stops a ClassCastException in the sorted-column pushdown on BigInteger, String, Instant and the other types it searches by ordering alone, and in the case-insensitive String filter. RangeFilter fails over for such a bound instead of casting it. Rejecting these values outright is left to the separate PR.
The Filter API.WhereFilterAdapter quotes a Character literal for a range filter, so FilterComparison.lt(X, Literal.of('5')) on a char column still means '5', not (char) 5.
Not changed
These knowingly differ from the query language:
A literal is read in the column's type.F == 0.1 on a float column means 0.1f, and X == -128 on a byte column means NULL_BYTE. The query language reads them as a double and an int. This is unchanged from main.
A value that converts exactly to the column's null value is null. An Integer -128 on a byte column, or a long Integer.MIN_VALUE on an int column, matches the null rows, as the literal -128 and the column type's own null value do, for ==, !=, in, direct values and range bounds. The query language compares it as a number below every value. This is unchanged from main. A char 65535 (NULL_CHAR) is the exception and is refused: it is the highest char, yet orders below every char.
Exact floating-point values on integral columns. A floating-point value that converts exactly to an int or long column, such as 2^24f or 2^53, matches only its exact equivalent. The query language's == also matches the integers that round to it.
A Float range bound on an int column converts to its exact int, and keeps the typed range filter. The query language compares the two in float, which rounds an int beyond 2^24, so X > 16777216f there excludes 16777217, which the converted bound includes. Failing over for every Float bound, even 5.0f, was too broad for a difference this rare.
Float and double range filters treat -0.0 and 0.0 as equal. The query language orders -0.0 below 0.0, so on rows holding -0.0, X < 0.0 and X >= 0.0 select other rows than it does.
Performance
A value that cannot be represented now costs a ConditionFilter, with no pushdown and no binary search, where main used a typed filter. Nearly every such query was answered wrongly on main. The exceptions are X == v on a double column with a long v beyond 2^53, where main's rounding to a double happens to agree with the query language's ==, and a range bound of another type, which threw ClassCastException on main.
Tests
MatchFilterCoercionTest checks each case against the ConditionFilter failover with assertSameRowsAsFailover, and pins the differences listed under "Not changed". It covers every primitive column type, BigInteger and BigDecimal columns, and char, null, NaN and -0.0 values, plus values of another type on sorted columns.
MatchFilterParamConversionTest covers the conversions themselves.
FailedOverFilterConsumersTest now forces a failover with 0x1p63 on a long column and new BigDecimal(1.1) on a double column.
Nonstandard Number implementations are treated as integers here, so lossy values can still pass conversion. For example, a DoubleAdder containing 5.7 has longValue() == 5; converting it for an int (or BigDecimal/BigInteger) column makes both sides compare as 5, so a direct MatchFilter matches 5 instead of rejecting the lossy value. Query-scope filters can also disagree with their formula failover. Please either reject unsupported Number subclasses or preserve and validate their fractional value rather than defaulting every non-float Number to longValue().
148 executable lines added or changed — 142/148 (95%) covered
Gaps
The one behavior this PR adds that no test drives is the range failover for an int column compared with a Float bound, which the query language compares in float; its conversion is exact, so only the explicit rule keeps it from selecting other rows, and nothing yet checks that rule against the failover. The rest is narrow: the unwrapping of a Python value, which these Java suites never pass; the error for an unquoted multi-letter literal on a char column; and, in RangeFilter, the array-value error and the two throws for a filter that cannot fail over, which count as changed only because conversionError was renamed.
Measured at f7193e961 against e998cc69c, the stack base (#8719 with #8709 merged in), so the numbers are this PR's own changes; #8709 and #8719 carry their own reports. Suites: test, testParallel, testSerial and testOutOfBand in :engine-table, and test and testOutOfBand in :extensions-parquet-table.
… is the converted value's decimal
The query language compares a float or double column with a BigDecimal or BigInteger through
BigDecimal.valueOf(double). Such a value now converts when it equals the converted value's
shortest decimal, as BigDecimal("0.1") does for 0.1, so it keeps the typed filter and `in`
accepts it. Any other value still fails over, or throws for `in`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…olumn
The query language, as Java, compares a char with a number by its code point. A Character
query-scope or direct value now converts that way for byte through double, BigInteger and
BigDecimal columns, so `X == c` keeps the typed filter, and `X in c`, a direct value and the
Filter API's eq(X, '5') no longer throw. A code point beyond a byte or short is rejected, so it
fails over, or throws where there is no failover.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eric column
A single-quoted char literal ('5') on a byte through double, BigInteger or BigDecimal column
now converts to its code point, as the query language compares it. `X == '5'` and `X < '5'`
keep the typed filter, `X in '5'` matches 53, and the Filter API's lt(X, '5'), whose range
filter has no failover, no longer throws.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t column
The int-column-vs-Float failover was too broad: it failed over for every Float bound, even
5.0f. A Float bound now converts to its exact int, as a large floating-point value does for
MatchFilter. The query language compares the two in float, so a bound beyond 2^24 can order a
neighbouring int differently; that is an accepted difference, documented in RangeFilter and
pinned by floatBoundOnAnIntColumnIsExact.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… fail over for such a range bound
- MatchFilter drops any match value that is neither null nor an instance of the column type, not only on a
BigDecimal column. Such a value matches nothing, but the sorted-column pushdown on BigInteger, String,
Instant and the other ordering-only types, and the case-insensitive String filter, threw
ClassCastException for it.
- NaN skips the convertor only on a primitive column, so the BigDecimal and BigInteger convertors reject it
and the filter fails over, rather than keeping a Double among the values of a sorted big-number column.
- RangeFilter fails over for a bound the convertor left as it is, instead of casting it.
- Drop RangeFilter's dead column-reference guard and MatchFilter.ColumnTypeConvertor.isColumnReference, and
unwrap a PyObject once in convertParam.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Non-standard Number instances containing NaN or infinity bypass the intended rejection because this branch returns before exactValue(a) validates the runtime type. For example, a DoubleAdder holding positive infinity narrows to Double.POSITIVE_INFINITY, compares equal here, and is accepted for a double column even though all nonstandard numeric types are documented as having no exact value. Validate the source type before the non-finite fast path so direct and in values are consistently refused.
Match query language parsing for leading-zero char literals
This parses leading-zero literals differently from the query language. For example, X == 010 is compiled by the ConditionFilter path as Java's octal code point 8, but Long.parseLong("010") produces 10, so the typed char filter selects a different row—the exact mismatch this change is meant to avoid. Decode Java-style decimal/octal notation here (or fail over for leading-zero forms).
…ue is null
An Integer -128 on a byte column, a long Integer.MIN_VALUE on an int column and so on now match the null rows, as the
column type's own null value and the literal -128 do, for ==, !=, in, direct values and range bounds, instead of being
rejected. The query language compares such a value as a number, so these differ from the ConditionFilter, as the
literal already does. A char 65535 (NULL_CHAR) and a value that only wraps onto the null value are still rejected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nonstandard Number implementations bypass rejection when their doubleValue() is NaN or infinity. For example, a DoubleAdder containing POSITIVE_INFINITY reaches this branch, compares equal to the narrowed infinity, and is accepted even though exactValue rejects the same type for finite values. Validate non-floating operands through exactValue before returning from the non-finite branch so all nonstandard numbers are consistently refused.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
lbooker42
added
NoDocumentationNeeded
ReleaseNotesNeeded
BREAKING CHANGE:
MatchFilterandRangeFilterno longer truncate, wrap or saturate a filter value to fit the column's type. Each change below replaces a result that disagreed with the query language, that is, with theConditionFilterthese filters fail over to.X in vwith a lossy query-scope value. Withv = 5.7on an int column,mainmatched 5.new MatchFilter(REGULAR, "X", 300)on a byte column matched 44 onmain.inoreqliteral, such asFilterIn.of(X, 4, 5.5).X == v,X != vand range filters with a lossy query-scope value fail over to the query language. SoX < vwithv = 5.7on an int column selects 4 and 5, wheremainselected 4.X == 5is(char) 5. WriteX == '5'for the digit.lt(X, '5')on an int column compares with 53, wheremainused 5.Split out of DH-23754. Rejecting values of the wrong type (
checkValueTypes) is left to a separate PR, because it reverses the "Will not implement" decision on DH-21232.The rule
MatchFilterandRangeFilterconverted a query-scope value to the column's type with the query language's casts, which truncate, wrap and saturate. Withv = 5.7on an int column,X == vmatched 5. Withv = 300on a byte column, it matched 44. The same condition written as a formula matches nothing.A value must now convert exactly, meaning the converted value selects exactly the rows the query language would. Otherwise it is refused:
X == v,X != vand range filters fail over to theirConditionFilter, which is always correct.X in v, direct values and Filter API values have no failover, so they throw.Changes
Numeric columns. A single
NumericColumnTypeConvertorhandles byte through double. It casts the value (narrow), thencheckRoundTripcompares the result with the original exactly, asBigDecimals. It also refuses:main, where the cast threwClassCastException;Numberof a nonstandard type, such asDoubleAdderorAtomicLong, which has no exact value to check.Chars. The query language, like Java, compares a char with a number by its code point. On numeric columns, byte through double,
BigIntegerandBigDecimal:Characterquery-scope or direct value converts to its code point, and so does a quoted char literal such as'5'.NULL_CHARis null.On a char column, a number converts to the char with that code point, and
NULL_CHAR(65535) is refused.BigDecimal and BigInteger columns convert exactly. A floating-point value converts through
BigDecimal.valueOf, as the query language compares it, and a long no longer goes throughdouble. NaN converts to neither, soX == vfails over, where NaN equals no big number, andX in vthrows.mainmatched 0 on aBigIntegercolumn.A BigDecimal or BigInteger on a float or double column. The query language compares these through
BigDecimal.valueOf(double), which uses the column value's shortest decimal representation. So the value converts only if it equals that decimal for its converted value:BigDecimal("0.1")converts to 0.1.new BigDecimal(0.1)is refused. It is exactly a double, but no double's shortest decimal equals it.BigDecimal("0.1")on a float column is refused, because 0.1f widens to 0.10000000149011612.Direct values, such as
new MatchFilter(options, "X", 5.7), now go through the same conversion. Before, they were used as given. WithoutnanMatch, NaN is now dropped from query-scope values on primitive columns, as it already was from direct values.Range bounds.
-0.0against a byte, short, int or char column fails over. The query language compares these withDouble.compare, which orders-0.0below0, so the converted bound0would select other rows. TheRangeFilterjavadoc documents this case, the remaining differences listed under "Not changed", and how null endpoints behave.Values of another type. The convertors pass a value they do not convert through as it is, such as a
Stringon aBigIntegercolumn. It can never match, soMatchFilternow drops it from its values on every column type, not onlyBigDecimal. That stops aClassCastExceptionin the sorted-column pushdown onBigInteger,String,Instantand the other types it searches by ordering alone, and in the case-insensitiveStringfilter.RangeFilterfails over for such a bound instead of casting it. Rejecting these values outright is left to the separate PR.The Filter API.
WhereFilterAdapterquotes aCharacterliteral for a range filter, soFilterComparison.lt(X, Literal.of('5'))on a char column still means'5', not(char) 5.Not changed
These knowingly differ from the query language:
F == 0.1on a float column means0.1f, andX == -128on a byte column meansNULL_BYTE. The query language reads them as a double and an int. This is unchanged frommain.Integer-128 on a byte column, or a longInteger.MIN_VALUEon an int column, matches the null rows, as the literal -128 and the column type's own null value do, for==,!=,in, direct values and range bounds. The query language compares it as a number below every value. This is unchanged frommain. A char 65535 (NULL_CHAR) is the exception and is refused: it is the highest char, yet orders below every char.2^24for2^53, matches only its exact equivalent. The query language's==also matches the integers that round to it.Floatrange bound on an int column converts to its exact int, and keeps the typed range filter. The query language compares the two in float, which rounds an int beyond 2^24, soX > 16777216fthere excludes 16777217, which the converted bound includes. Failing over for everyFloatbound, even5.0f, was too broad for a difference this rare.-0.0and0.0as equal. The query language orders-0.0below0.0, so on rows holding-0.0,X < 0.0andX >= 0.0select other rows than it does.Performance
A value that cannot be represented now costs a
ConditionFilter, with no pushdown and no binary search, wheremainused a typed filter. Nearly every such query was answered wrongly onmain. The exceptions areX == von a double column with a longvbeyond 2^53, wheremain's rounding to a double happens to agree with the query language's==, and a range bound of another type, which threwClassCastExceptiononmain.Tests
MatchFilterCoercionTestchecks each case against theConditionFilterfailover withassertSameRowsAsFailover, and pins the differences listed under "Not changed". It covers every primitive column type,BigIntegerandBigDecimalcolumns, and char, null, NaN and-0.0values, plus values of another type on sorted columns.MatchFilterParamConversionTestcovers the conversions themselves.FailedOverFilterConsumersTestnow forces a failover with0x1p63on a long column andnew BigDecimal(1.1)on a double column.🤖 Generated with Claude Code