[Vector Upsert 4/5] Apply the query's visible-document set before vector candidate generation - #19300
Merged
xiangfu0 merged 1 commit intoAug 27, 2026
Conversation
This was referenced Aug 19, 2026
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #19300 +/- ##
============================================
+ Coverage 67.47% 67.53% +0.05%
Complexity 1430 1430
============================================
Files 3486 3486
Lines 223985 224043 +58
Branches 35340 35353 +13
============================================
+ Hits 151138 151307 +169
+ Misses 60818 60704 -114
- Partials 12029 12032 +3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
xiangfu0
force-pushed
the
xiangfu0/upsert-vector-4-upsert-candidate-scope
branch
from
August 19, 2026 01:38
ca0c7f0 to
7ee68b3
Compare
xiangfu0
changed the title
xiangfu0
added
bug
xiangfu0
force-pushed
the
xiangfu0/upsert-vector-4-upsert-candidate-scope
branch
9 times, most recently
from
August 26, 2026 21:32
4ee706e to
09c3336
Compare
…tor candidate generation FilterPlanNode constructed and executed VECTOR_SIMILARITY before adding SegmentContext.getDocIdsSnapshot() as an outer AND. Obsolete physical versions of upserted rows could therefore occupy per-segment ANN top-K slots and be removed only afterward, producing fewer than K rows or omitting nearer current rows. Pass the snapshot to vector predicates as a required VectorCandidateScope so it constrains candidate generation, while retaining the outer bitmap AND as defense in depth. FilterPlanNode remains the only place that knows the scope comes from the segment's queryable-document snapshot; the operators only see 'documents this predicate may consider'. Choose the execution path at plan time, where both the reader capability and forward index availability are known: - a filter-aware reader receives the scope and does filtered ANN; - a reader that cannot restrict its search is bypassed for ExactVectorScanFilterOperator over the allowed documents, which reports it through fallbackReason; - an empty scope needs no candidate generation and no reader capability at all, so it short-circuits to an empty operator; - when neither path is available the query fails clearly. Because the choice is made once, VectorSimilarityFilterOperator no longer needs a runtime exact-scan branch, and it captures the reader capability once instead of re-reading it per execution, so a plan built on one answer can never execute against another. Required scopes stay separate from optimizer-selected metadata filters and are intersected with them before candidate generation. Non-upsert queries keep the adaptive metadata behavior, and non-vector plans do not copy the snapshot.
xiangfu0
force-pushed
the
xiangfu0/upsert-vector-4-upsert-candidate-scope
branch
from
August 26, 2026 23:34
09c3336 to
1b3f7bd
Compare
Jackie-Jiang
approved these changes
Aug 27, 2026
xiangfu0
added a commit
to pinot-contrib/pinot-docs
that referenced
this pull request
Aug 27, 2026
Documents how Pinot constrains vector candidate generation to the queryable-document snapshot for upsert and deleted rows, including filtered-ANN selection, exact-scan fallback cost, failure behavior, and EXPLAIN attributes. Follow-up to apache/pinot#19300. Co-authored-by: Xiang Fu <xiangfu@Xiang-mac-mtv-2.local>
Contributor
Author
|
Documentation follow-up: pinot-contrib/pinot-docs#1011 (merged). It documents queryable-document scoping before vector candidate generation, filtered-ANN versus exact-scan behavior, fallback cost, failure semantics, and the new EXPLAIN attributes. |
This was referenced Aug 27, 2026
xiangfu0
added a commit
to xiangfu0/pinot
that referenced
this pull request
Aug 27, 2026
…lter query Builds on the FULL-upsert candidate-generation fix now in master (apache#19297-apache#19300), which selects filtered ANN whenever a reader advertises it. This makes the mutable index one of those readers, and tightens the contract that decision rests on. - Make MutableVectorIndex filter-aware: store the supplied Pinot document ID as a numeric doc value alongside the existing stored field, translate hits through it, and report supportsPreFilter()=true. Consuming segments of upsert tables therefore use filtered ANN instead of falling back to an exact scan. - Extract BasePinotDocIdBitmapFilterQuery so the mutable index and HnswVectorIndexReader share one Lucene filter iterator instead of keeping two copies (HnswVectorIndexReader.RoaringBitmapFilterQuery now extends it). - Turn FilterAwareVectorIndexReader#supportsPreFilter into a hard contract: a reader returning true must always return a strict subset of the supplied bitmap and must never heuristically degrade to unfiltered search, because the engine relies on filtered search to keep obsolete row versions out of candidate generation. Readers that can only honor the filter conditionally must return false and take the exact-scan fallback. - Restrict VECTOR_SIMILARITY_RADIUS candidates to the same document set and disable the approximate radius path when it applies. Radius results were already correct without this (the operator falls back to a complete brute-force scan when its candidate pool saturates, and the outer snapshot AND removes obsolete matches), so this avoids spending the candidate budget, and triggering the expensive saturation fallback, on rows the query cannot see.
xiangfu0
added a commit
to xiangfu0/pinot
that referenced
this pull request
Aug 27, 2026
…lter query Builds on the FULL-upsert candidate-generation fix now in master (apache#19297-apache#19300), which selects filtered ANN whenever a reader advertises it. This makes the mutable index one of those readers, and tightens the contract that decision rests on. - Make MutableVectorIndex filter-aware: store the supplied Pinot document ID as a numeric doc value alongside the existing stored field, translate hits through it, and report supportsPreFilter()=true. Consuming segments of upsert tables therefore use filtered ANN instead of falling back to an exact scan. - Extract BasePinotDocIdBitmapFilterQuery so the mutable index and HnswVectorIndexReader share one Lucene filter iterator instead of keeping two copies (HnswVectorIndexReader.RoaringBitmapFilterQuery now extends it). - Turn FilterAwareVectorIndexReader#supportsPreFilter into a hard contract: a reader returning true must always return a strict subset of the supplied bitmap and must never heuristically degrade to unfiltered search, because the engine relies on filtered search to keep obsolete row versions out of candidate generation. Readers that can only honor the filter conditionally must return false and take the exact-scan fallback. - Restrict VECTOR_SIMILARITY_RADIUS candidates to the same document set and disable the approximate radius path when it applies. Radius results were already correct without this (the operator falls back to a complete brute-force scan when its candidate pool saturates, and the outer snapshot AND removes obsolete matches), so this avoids spending the candidate budget, and triggering the expensive saturation fallback, on rows the query cannot see.
xiangfu0
added a commit
to xiangfu0/pinot
that referenced
this pull request
Aug 27, 2026
…lter query The FULL-upsert candidate-generation fix (apache#19297-apache#19300) makes FilterPlanNode select filtered ANN whenever a reader advertises the capability, and fall back to a correctness-first exact scan when it does not. Consuming segments of upsert tables are left on that fallback because the mutable index cannot restrict its search. This makes it one of the readers that can. - MutableVectorIndex stores the supplied Pinot document ID as a numeric doc value alongside the existing stored field, translates hits through it, and reports supportsPreFilter()=true, so consuming segments use filtered ANN. No planner change is needed: master already routes on that capability. - Extract BasePinotDocIdBitmapFilterQuery so the mutable index and HnswVectorIndexReader share one Lucene filter iterator instead of two copies (HnswVectorIndexReader.RoaringBitmapFilterQuery now extends it). - Make FilterAwareVectorIndexReader#supportsPreFilter a hard contract: a reader returning true must always return a strict subset of the supplied bitmap and must never heuristically degrade to unfiltered search, because the engine relies on filtered search to keep obsolete row versions out of candidate generation. Readers that can only honor the filter conditionally must return false and take the exact-scan fallback. The previous wording invited exactly the conditional behavior this forbids. - VectorUpsertTableTest asserted that consuming segments fall back to the exact scan because the mutable index was not filter-aware. That is what this changes, so those assertions now require filtered ANN there.
xiangfu0
added a commit
to xiangfu0/pinot
that referenced
this pull request
Aug 27, 2026
…lter query The FULL-upsert candidate-generation fix (apache#19297-apache#19300) makes FilterPlanNode select filtered ANN whenever a reader advertises the capability, and fall back to a correctness-first exact scan when it does not. Consuming segments of upsert tables are left on that fallback because the mutable index cannot restrict its search. This makes it one of the readers that can. - MutableVectorIndex stores the supplied Pinot document ID as a numeric doc value, drives both filtered traversal and hit translation from it, and reports supportsPreFilter()=true, so consuming segments use filtered ANN. No planner change is needed: master already routes on that capability. Filtered search refreshes a shared near-real-time searcher so it sees every added row, committed or not; the unfiltered path keeps the cheaper last-committed view. - Give each index instance a private directory under a segment-named parent, so two replicas of one segment hosted in the same JVM cannot collide on the Lucene write lock, while segment and column stay in the path for diagnostics. - Extract BasePinotDocIdBitmapFilterQuery so the mutable index and HnswVectorIndexReader share one Lucene filter iterator instead of two copies (HnswVectorIndexReader.RoaringBitmapFilterQuery now extends it). - Make FilterAwareVectorIndexReader#supportsPreFilter a hard contract: a reader returning true must always return a strict subset of the supplied bitmap and must never heuristically degrade to unfiltered search, because the engine relies on filtered search to keep obsolete row versions out of candidate generation. Readers that can only honor the filter conditionally must return false and take the exact-scan fallback. - VectorUpsertTableTest asserted that consuming segments fall back to the exact scan because the mutable index was not filter-aware. That is what this changes, so those assertions now require filtered ANN there.
xiangfu0
added a commit
to xiangfu0/pinot
that referenced
this pull request
Aug 27, 2026
…lter query The FULL-upsert candidate-generation fix (apache#19297-apache#19300) makes FilterPlanNode select filtered ANN whenever a reader advertises the capability, and fall back to a correctness-first exact scan when it does not. Consuming segments of upsert tables are left on that fallback because the mutable index cannot restrict its search. This makes it one of the readers that can. - MutableVectorIndex stores the supplied Pinot document ID as a numeric doc value, drives both filtered traversal and hit translation from it, and reports supportsPreFilter()=true, so consuming segments use filtered ANN. No planner change is needed: master already routes on that capability. Filtered search refreshes a shared near-real-time searcher so it sees every added row, committed or not; the unfiltered path keeps the cheaper last-committed view. - Give each index instance a private directory under a segment-named parent, so two replicas of one segment hosted in the same JVM cannot collide on the Lucene write lock, while segment and column stay in the path for diagnostics. - Extract BasePinotDocIdBitmapFilterQuery so the mutable index and HnswVectorIndexReader share one Lucene filter iterator instead of two copies (HnswVectorIndexReader.RoaringBitmapFilterQuery now extends it). - Make FilterAwareVectorIndexReader#supportsPreFilter a hard contract: a reader returning true must always return a strict subset of the supplied bitmap and must never heuristically degrade to unfiltered search, because the engine relies on filtered search to keep obsolete row versions out of candidate generation. Readers that can only honor the filter conditionally must return false and take the exact-scan fallback. - VectorUpsertTableTest asserted that consuming segments fall back to the exact scan because the mutable index was not filter-aware. That is what this changes, so those assertions now require filtered ANN there.
xiangfu0
added a commit
to xiangfu0/pinot
that referenced
this pull request
Aug 27, 2026
…lter query The FULL-upsert candidate-generation fix (apache#19297-apache#19300) makes FilterPlanNode select filtered ANN whenever a reader advertises the capability, and fall back to a correctness-first exact scan when it does not. Consuming segments of upsert tables are left on that fallback because the mutable index cannot restrict its search. This makes it one of the readers that can. - MutableVectorIndex stores the supplied Pinot document ID as a numeric doc value, drives both filtered traversal and hit translation from it, and reports supportsPreFilter()=true, so consuming segments use filtered ANN. No planner change is needed: master already routes on that capability. Filtered search refreshes a shared near-real-time searcher so it sees every added row, committed or not; the unfiltered path keeps the cheaper last-committed view. - Give each index instance a private directory under a segment-named parent, so two replicas of one segment hosted in the same JVM cannot collide on the Lucene write lock, while segment and column stay in the path for diagnostics. - Extract BasePinotDocIdBitmapFilterQuery so the mutable index and HnswVectorIndexReader share one Lucene filter iterator instead of two copies (HnswVectorIndexReader.RoaringBitmapFilterQuery now extends it). - Make FilterAwareVectorIndexReader#supportsPreFilter a hard contract: a reader returning true must always return a strict subset of the supplied bitmap and must never heuristically degrade to unfiltered search, because the engine relies on filtered search to keep obsolete row versions out of candidate generation. Readers that can only honor the filter conditionally must return false and take the exact-scan fallback. - VectorUpsertTableTest asserted that consuming segments fall back to the exact scan because the mutable index was not filter-aware. That is what this changes, so those assertions now require filtered ANN there.
xiangfu0
added a commit
to xiangfu0/pinot
that referenced
this pull request
Aug 27, 2026
…lter query The FULL-upsert candidate-generation fix (apache#19297-apache#19300) makes FilterPlanNode select filtered ANN whenever a reader advertises the capability, and fall back to a correctness-first exact scan when it does not. Consuming segments of upsert tables are left on that fallback because the mutable index cannot restrict its search. This makes it one of the readers that can. - MutableVectorIndex stores the supplied Pinot document ID as a numeric doc value, drives both filtered traversal and hit translation from it, and reports supportsPreFilter()=true, so consuming segments use filtered ANN. No planner change is needed: master already routes on that capability. Filtered search refreshes a shared near-real-time searcher so it sees every added row, committed or not; the unfiltered path keeps the cheaper last-committed view. - Give each index instance a private directory under a segment-named parent, so two replicas of one segment hosted in the same JVM cannot collide on the Lucene write lock, while segment and column stay in the path for diagnostics. - Extract BasePinotDocIdBitmapFilterQuery so the mutable index and HnswVectorIndexReader share one Lucene filter iterator instead of two copies (HnswVectorIndexReader.RoaringBitmapFilterQuery now extends it). - Make FilterAwareVectorIndexReader#supportsPreFilter a hard contract: a reader returning true must always return a strict subset of the supplied bitmap and must never heuristically degrade to unfiltered search, because the engine relies on filtered search to keep obsolete row versions out of candidate generation. Readers that can only honor the filter conditionally must return false and take the exact-scan fallback. - VectorUpsertTableTest asserted that consuming segments fall back to the exact scan because the mutable index was not filter-aware. That is what this changes, so those assertions now require filtered ANN there.
xiangfu0
added a commit
to xiangfu0/pinot
that referenced
this pull request
Aug 28, 2026
…lter query The FULL-upsert candidate-generation fix (apache#19297-apache#19300) makes FilterPlanNode select filtered ANN whenever a reader advertises the capability, and fall back to a correctness-first exact scan when it does not. Consuming segments of upsert tables are left on that fallback because the mutable index cannot restrict its search. This makes it one of the readers that can. - MutableVectorIndex stores the supplied Pinot document ID as a numeric doc value, drives both filtered traversal and hit translation from it, and reports supportsPreFilter()=true, so consuming segments use filtered ANN. No planner change is needed: master already routes on that capability. Filtered search refreshes a shared near-real-time searcher so it sees every added row, committed or not; the unfiltered path keeps the cheaper last-committed view. - Give each index instance a private directory under a segment-named parent, so two replicas of one segment hosted in the same JVM cannot collide on the Lucene write lock, while segment and column stay in the path for diagnostics. - Extract BasePinotDocIdBitmapFilterQuery so the mutable index and HnswVectorIndexReader share one Lucene filter iterator instead of two copies (HnswVectorIndexReader.RoaringBitmapFilterQuery now extends it). - Make FilterAwareVectorIndexReader#supportsPreFilter a hard contract: a reader returning true must always return a strict subset of the supplied bitmap and must never heuristically degrade to unfiltered search, because the engine relies on filtered search to keep obsolete row versions out of candidate generation. Readers that can only honor the filter conditionally must return false and take the exact-scan fallback. - VectorUpsertTableTest asserted that consuming segments fall back to the exact scan because the mutable index was not filter-aware. That is what this changes, so those assertions now require filtered ANN there.
xiangfu0
added a commit
to xiangfu0/pinot
that referenced
this pull request
Aug 29, 2026
…lter query The FULL-upsert candidate-generation fix (apache#19297-apache#19300) makes FilterPlanNode select filtered ANN whenever a reader advertises the capability, and fall back to a correctness-first exact scan when it does not. Consuming segments of upsert tables are left on that fallback because the mutable index cannot restrict its search. This makes it one of the readers that can. - MutableVectorIndex stores the supplied Pinot document ID as a numeric doc value, drives both filtered traversal and hit translation from it, and reports supportsPreFilter()=true, so consuming segments use filtered ANN. No planner change is needed: master already routes on that capability. Filtered search refreshes a shared near-real-time searcher so it sees every added row, committed or not; the unfiltered path keeps the cheaper last-committed view. - Give each index instance a private directory under a segment-named parent, so two replicas of one segment hosted in the same JVM cannot collide on the Lucene write lock, while segment and column stay in the path for diagnostics. - Extract BasePinotDocIdBitmapFilterQuery so the mutable index and HnswVectorIndexReader share one Lucene filter iterator instead of two copies (HnswVectorIndexReader.RoaringBitmapFilterQuery now extends it). - Make FilterAwareVectorIndexReader#supportsPreFilter a hard contract: a reader returning true must always return a strict subset of the supplied bitmap and must never heuristically degrade to unfiltered search, because the engine relies on filtered search to keep obsolete row versions out of candidate generation. Readers that can only honor the filter conditionally must return false and take the exact-scan fallback. - VectorUpsertTableTest asserted that consuming segments fall back to the exact scan because the mutable index was not filter-aware. That is what this changes, so those assertions now require filtered ANN there.
xiangfu0
added a commit
to xiangfu0/pinot
that referenced
this pull request
Aug 29, 2026
…lter query The FULL-upsert candidate-generation fix (apache#19297-apache#19300) makes FilterPlanNode select filtered ANN whenever a reader advertises the capability, and fall back to a correctness-first exact scan when it does not. Consuming segments of upsert tables are left on that fallback because the mutable index cannot restrict its search. This makes it one of the readers that can. - MutableVectorIndex stores the supplied Pinot document ID as a numeric doc value, drives both filtered traversal and hit translation from it, and reports supportsPreFilter()=true, so consuming segments use filtered ANN. No planner change is needed: master already routes on that capability. Filtered search refreshes a shared near-real-time searcher so it sees every added row, committed or not; the unfiltered path keeps the cheaper last-committed view. - Give each index instance a private directory under a segment-named parent, so two replicas of one segment hosted in the same JVM cannot collide on the Lucene write lock, while segment and column stay in the path for diagnostics. - Extract BasePinotDocIdBitmapFilterQuery so the mutable index and HnswVectorIndexReader share one Lucene filter iterator instead of two copies (HnswVectorIndexReader.RoaringBitmapFilterQuery now extends it). - Make FilterAwareVectorIndexReader#supportsPreFilter a hard contract: a reader returning true must always return a strict subset of the supplied bitmap and must never heuristically degrade to unfiltered search, because the engine relies on filtered search to keep obsolete row versions out of candidate generation. Readers that can only honor the filter conditionally must return false and take the exact-scan fallback. - VectorUpsertTableTest asserted that consuming segments fall back to the exact scan because the mutable index was not filter-aware. That is what this changes, so those assertions now require filtered ANN there.
xiangfu0
added a commit
to xiangfu0/pinot
that referenced
this pull request
Aug 29, 2026
…lter query The FULL-upsert candidate-generation fix (apache#19297-apache#19300) makes FilterPlanNode select filtered ANN whenever a reader advertises the capability, and fall back to a correctness-first exact scan when it does not. Consuming segments of upsert tables are left on that fallback because the mutable index cannot restrict its search. This makes it one of the readers that can. - MutableVectorIndex stores the supplied Pinot document ID as a numeric doc value, drives both filtered traversal and hit translation from it, and reports supportsPreFilter()=true, so consuming segments use filtered ANN. No planner change is needed: master already routes on that capability. Filtered search refreshes a shared near-real-time searcher so it sees every added row, committed or not; the unfiltered path keeps the cheaper last-committed view. - Give each index instance a private directory under a segment-named parent, so two replicas of one segment hosted in the same JVM cannot collide on the Lucene write lock, while segment and column stay in the path for diagnostics. - Extract BasePinotDocIdBitmapFilterQuery so the mutable index and HnswVectorIndexReader share one Lucene filter iterator instead of two copies (HnswVectorIndexReader.RoaringBitmapFilterQuery now extends it). - Make FilterAwareVectorIndexReader#supportsPreFilter a hard contract: a reader returning true must always return a strict subset of the supplied bitmap and must never heuristically degrade to unfiltered search, because the engine relies on filtered search to keep obsolete row versions out of candidate generation. Readers that can only honor the filter conditionally must return false and take the exact-scan fallback. - VectorUpsertTableTest asserted that consuming segments fall back to the exact scan because the mutable index was not filter-aware. That is what this changes, so those assertions now require filtered ANN there.
xiangfu0
added a commit
to xiangfu0/pinot
that referenced
this pull request
Aug 30, 2026
…lter query The FULL-upsert candidate-generation fix (apache#19297-apache#19300) makes FilterPlanNode select filtered ANN whenever a reader advertises the capability, and fall back to a correctness-first exact scan when it does not. Consuming segments of upsert tables are left on that fallback because the mutable index cannot restrict its search. This makes it one of the readers that can. - MutableVectorIndex stores the supplied Pinot document ID as a numeric doc value, drives both filtered traversal and hit translation from it, and reports supportsPreFilter()=true, so consuming segments use filtered ANN. No planner change is needed: master already routes on that capability. Filtered search refreshes a shared near-real-time searcher so it sees every added row, committed or not; the unfiltered path keeps the cheaper last-committed view. - Give each index instance a private directory under a segment-named parent, so two replicas of one segment hosted in the same JVM cannot collide on the Lucene write lock, while segment and column stay in the path for diagnostics. - Extract BasePinotDocIdBitmapFilterQuery so the mutable index and HnswVectorIndexReader share one Lucene filter iterator instead of two copies (HnswVectorIndexReader.RoaringBitmapFilterQuery now extends it). - Make FilterAwareVectorIndexReader#supportsPreFilter a hard contract: a reader returning true must always return a strict subset of the supplied bitmap and must never heuristically degrade to unfiltered search, because the engine relies on filtered search to keep obsolete row versions out of candidate generation. Readers that can only honor the filter conditionally must return false and take the exact-scan fallback. - VectorUpsertTableTest asserted that consuming segments fall back to the exact scan because the mutable index was not filter-aware. That is what this changes, so those assertions now require filtered ANN there.
xiangfu0
added a commit
to xiangfu0/pinot
that referenced
this pull request
Aug 30, 2026
…lter query The FULL-upsert candidate-generation fix (apache#19297-apache#19300) makes FilterPlanNode select filtered ANN whenever a reader advertises the capability, and fall back to a correctness-first exact scan when it does not. Consuming segments of upsert tables are left on that fallback because the mutable index cannot restrict its search. This makes it one of the readers that can. - MutableVectorIndex stores the supplied Pinot document ID as a numeric doc value, drives both filtered traversal and hit translation from it, and reports supportsPreFilter()=true, so consuming segments use filtered ANN. No planner change is needed: master already routes on that capability. Filtered search refreshes a shared near-real-time searcher so it sees every added row, committed or not; the unfiltered path keeps the cheaper last-committed view. - Give each index instance a private directory under a segment-named parent, so two replicas of one segment hosted in the same JVM cannot collide on the Lucene write lock, while segment and column stay in the path for diagnostics. - Extract BasePinotDocIdBitmapFilterQuery so the mutable index and HnswVectorIndexReader share one Lucene filter iterator instead of two copies (HnswVectorIndexReader.RoaringBitmapFilterQuery now extends it). - Make FilterAwareVectorIndexReader#supportsPreFilter a hard contract: a reader returning true must always return a strict subset of the supplied bitmap and must never heuristically degrade to unfiltered search, because the engine relies on filtered search to keep obsolete row versions out of candidate generation. Readers that can only honor the filter conditionally must return false and take the exact-scan fallback. - VectorUpsertTableTest asserted that consuming segments fall back to the exact scan because the mutable index was not filter-aware. That is what this changes, so those assertions now require filtered ANN there.
xiangfu0
added a commit
to xiangfu0/pinot
that referenced
this pull request
Aug 30, 2026
…lter query The FULL-upsert candidate-generation fix (apache#19297-apache#19300) makes FilterPlanNode select filtered ANN whenever a reader advertises the capability, and fall back to a correctness-first exact scan when it does not. Consuming segments of upsert tables are left on that fallback because the mutable index cannot restrict its search. This makes it one of the readers that can. - MutableVectorIndex stores the supplied Pinot document ID as a numeric doc value, drives both filtered traversal and hit translation from it, and reports supportsPreFilter()=true, so consuming segments use filtered ANN. No planner change is needed: master already routes on that capability. Filtered search refreshes a shared near-real-time searcher so it sees every added row, committed or not; the unfiltered path keeps the cheaper last-committed view. - Give each index instance a private directory under a segment-named parent, so two replicas of one segment hosted in the same JVM cannot collide on the Lucene write lock, while segment and column stay in the path for diagnostics. - Extract BasePinotDocIdBitmapFilterQuery so the mutable index and HnswVectorIndexReader share one Lucene filter iterator instead of two copies (HnswVectorIndexReader.RoaringBitmapFilterQuery now extends it). - Make FilterAwareVectorIndexReader#supportsPreFilter a hard contract: a reader returning true must always return a strict subset of the supplied bitmap and must never heuristically degrade to unfiltered search, because the engine relies on filtered search to keep obsolete row versions out of candidate generation. Readers that can only honor the filter conditionally must return false and take the exact-scan fallback. - VectorUpsertTableTest asserted that consuming segments fall back to the exact scan because the mutable index was not filter-aware. That is what this changes, so those assertions now require filtered ANN there.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 4/5 of the split of #19287 (Fix FULL-upsert vector candidate generation).
This is the core fix. Stacked on #19299.
Summary
FilterPlanNodeconstructed and executedVECTOR_SIMILARITYbefore addingSegmentContext.getDocIdsSnapshot()as an outer AND. Obsolete physical versions ofupserted rows could therefore occupy per-segment ANN top-K slots and be removed only
afterward, producing fewer than K rows or omitting nearer current rows.
Pass the snapshot to vector predicates as a required
VectorCandidateScopeso itconstrains candidate generation, while retaining the outer bitmap AND as defense in
depth.
FilterPlanNoderemains the only place that knows the scope comes from thesegment's queryable-document snapshot; the operators only ever see "documents this
predicate may consider". That snapshot is already more general than upsert — it is the
queryable-document set, which also covers delete tombstones and
skipUpsertDelete— sono upsert vocabulary leaks into
pinot-core's filter operators.The execution path is chosen at plan time, where both the reader capability and
forward index availability are known:
ExactVectorScanFilterOperatorover the allowed documents, reported throughfallbackReason;short-circuits to an empty operator;
Because the choice is made once,
VectorSimilarityFilterOperatorneeds no runtimeexact-scan branch, and it captures the reader capability once instead of re-reading it
per execution, so a plan built on one answer can never execute against another. Required
scopes stay separate from optimizer-selected metadata filters and are intersected with
them before candidate generation. Non-upsert queries keep the adaptive metadata
behavior, non-vector plans do not copy the snapshot, and the new explain attributes are
emitted only for queries that actually carry a scope.
Performance and compatibility
ANN approximation semantics are unchanged. Segments whose reader cannot restrict its
search use a correctness-first exact scan whose cost is proportional to the
allowed-document count times vector dimension; consuming segments of upsert tables take
this path today because
MutableVectorIndexis not filter-aware. Making the mutableindex filter-aware would retire that fallback and can be done separately.
Validation
(
FilterPlanNodeTest,VectorSimilarityFilterOperatorTest,FilterAwareVectorSearchTest,ExactVectorScanFilterOperatorTest,VectorRadiusFilterOperatorTest,VectorSearchStrategyTest).Stack