Repository navigation
feat(java): make a declined pushdown visible instead of silent - #972
Conversation
|
cc @SemyonSinchenko @SYaoJun — stacked on #971 (shows its commits until that merges, diff will shrink automatically). Green CI. |
|
@SemyonSinchenko @SYaoJun this one builds on #971, so please review that first. It is opened against The slice itself makes a declined pushdown observable: a backend reports which hints it applied, and declining one must not change the results. |
301fc00 to
a91cefa
Compare
|
Rebased onto current This one sits on top of #971 — reviewing and merging #971 first leaves a single commit here. Together the two are the last piece the Parquet backend in #964 needs, so I am holding that PR until both land rather than opening a dependent chain. |
|
@SemyonSinchenko this is the other half of the read contract — a backend reports which hints it actually applied, so a declined pushdown is visible rather than silently changing nothing. Green and rebased on current |
A physical read is not only a URI. A caller knows which columns it needs, which source rows, which values, and how many rows are enough - and a backend that learns all four up front can skip work that a caller would otherwise have to discard after the fact. Add the request half of the read API: an ordered projection, a half-open row range, an AND conjunction of row predicates over immutable scalars, and a row limit. A request also reports which of those hints it actually carries, so a backend can answer for each one instead of guessing. A column is addressed through ColumnRef, not a bare String. A reference is unresolved by construction because a request exists before its file is opened; a backend binds it with Schema.resolve, which matches the exact name and refuses an unknown or ambiguous column instead of picking the first hit. A Filter is a closed predicate tree: a column comparison, or AND, OR and NOT over predicates, built through one named factory per operator so a null check cannot be handed a value and a comparison cannot be handed null. A backend walks the tree through a Visitor with one callback per node kind, so adding a node kind is a compile error in every reader rather than a silently skipped branch. Constraint: filter literals stay immutable and lossless. NaN, infinity, and sub-millisecond timestamps are refused rather than silently compared against a value a stored format cannot represent.
A backend that cannot apply a hint must still return the same rows as one that can. That leaves a caller unable to tell an applied filter from an ignored one, which matters: an ignored row range or limit changes cost, not results. Add the reader half of the read API. A read returns a cursor together with a report that names every requested hint as applied or declined, and the result refuses to be built unless the report accounts for exactly the hints the request carried - no gaps, no invented ones. Directive: declining a pushdown must not change the rows or the schema a read returns. The contract test asserts that a reader declining everything produces the same output as one applying everything.
a91cefa to
f3a4ff1
Compare
|
Addressed every thread above. Because the files live in the #971 commit this PR stacks on, the changes landed there ( |
|
Merging this because there are no risks: Java module is not published yet. |
A projection and a filter used to reference a column by a bare string and a filter was a single comparison; a backend could not tell a name from a qualified path and a caller could not express AND/OR/NOT. The read request now carries ColumnRef, a typed exact-name reference that only a Schema resolves, and Filter is a closed predicate tree of comparisons, junctions and negation walked through a Visitor. Every reader, Parquet and Iceberg consumer builds its projection from ColumnRef; the Parquet backend still declines filter pushdown, so results are unchanged. Constraint: mirrors apache#971 and apache#972 so the fork stays source-compatible with the upstream io-api surface.
2 participants
Adds the reader side:
PhysicalReader,ReadReport,ReadResult, plus a contract test that pins the invariant a physical reader must hold — a reader that declines every pushdown hint returns the same rows and the same schema as one that applies all of them. Only the reportedapplied/declinedsets differ.Builds on #971 (this PR's base branch). Together with the merged #961 (batch/write shapes) and #971 (request shapes), this completes the read half of the IO API.
Focused verify:
mvn --no-transfer-progress -f maven-projects/pom.xml -pl io-api -am spotless:checkandclean verify -Dspotless.check.skip=true— clean, 63 tests green.