[4.2] feat: backport FULLTEXT2 from main

[VioletQwQ-0]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

issue #25541

What this PR does / why we need it:

This is a bounded backport of the FULLTEXT2 behavior closure from the fixed
upstream/main reference to 4.2-dev. It keeps FULLTEXT2 distinct from
classic FULLTEXT and preserves the target branch's parser/PERFORM/pipe
contracts.

Source/target closure:

• source upstream/main: 8e616a693458652cc26b3c4be45c6602dff298d4
• target upstream/4.2-dev: d2393868a7aaa6343518d80849fe4696aa3577e9
• exact fork head: 325686d29c17b3729f32b837b5bc2144d33f0a9b
• feat: fulltext2 — complete rewrite of fulltext index and add a position-free BM25 full-text retrieval index #25904 core: 966acb2a92bb740e57b0b7a181df2d21956e1d3, 0eb9a921ece274c956447640b41ad2a255079e8d, 403a99fb65304e75b82e75793448f0690a871d0e, bfa83f42737ca6304c0af3f6913e70ccc961464
• perf(fulltext2): decode phrase positions after doc alignment #27165 phrase-position decode: 2b142144c3c24a2100d7d313e952717025d56aeb
• FIX: CREATE/ALTER/REPLACE VIEW persists unusable MATCH queries without a matching FULLTEXT index #27171 FT2-only wrapped score/range closure: 835e046faa9242dbe288bd72364a475e2132bbed
• perf(fulltext2): add cold-load observability #27461/perf(fulltext2): reuse immutable bases across cache generations #27462 cache and cold-load closure: e4abf5e90d0b5e09590571ca78447e50806527c
• test: replace fulltext2 BVT sleeps with readiness waits #27598 readiness BVT closure: e3a7440e2ac32bac780669f33c656960e22cff22

Current-head review repairs:

• Reverse constant-left </<= before harvesting a lower-bound MATCH
  membership driver; reject constant-left >/>= upper bounds fail-closed.
• Route all FULLTEXT2 ISCP consumer errors through the non-blocking helper and
  report ctx.Err() on cancellation.
• Restore the execution-level fulltext2_wrapped_score.sql/.result BVT pair
  and replace restore/clone fixed sleeps with readiness waits.
• Refresh the ancestor-accurate backport manifest and clear all
  git diff --check trailing whitespace findings.
• Trim FULLTEXT2 SHOW CREATE option padding only at the terminal output
  boundary, preserving the shared renderer and vector-index behavior.
• Route wrapped-only FULLTEXT2 score predicates on either INNER JOIN child
  through the existing wrapped rewrite and keep the score comparison in a
  post-rewrite FILTER.

Validation

• make -C pkg/sql/parsers/dialect/mysql — PASS
• make generate-pb — PASS; protoc 29.3 descriptor-only churn was discarded
  and no unexpected generated diff remains
• go test -mod=readonly ./pkg/fulltext2/... ./pkg/iscp ./pkg/sql/plan — PASS
• current-head focused GOWORK=off go test -mod=readonly -count=1 -timeout 120s -run '^TestFulltext2WrappedMatchJoinChild$' ./pkg/sql/plan — PASS
• current-head GOWORK=off go test -mod=readonly -count=1 ./pkg/sql/plan — PASS
• current-head GOWORK=off go build -mod=readonly ./pkg/sql/plan and
  GOWORK=off go vet -mod=readonly ./pkg/sql/plan — PASS
• current-head SCA repair — removed the unreferenced applyJoinFullTextIndices
  wrapper; targeted plan tests, build, and vet pass; exact-head remote SCA is
  pending
• current-head focused SHOW CREATE regression and catalog option tests — PASS
• local mo-cgo-test wrapper — BLOCKED by missing cgo/libmo.dylib; no CGo
  wrapper pass is claimed
• go test -race -mod=readonly ./pkg/fulltext2/... ./pkg/iscp — PASS
• go build -mod=readonly ./cmd/mo-service — PASS (macOS linker rpath/path
  warnings only)
• base differential — exact target d2393868a7aaa6343518d80849fe4696aa3577e9
  rejects CREATE FULLTEXT2 INDEX; this exact head parses it as
  INDEX_CATEGORY_FULLTEXT2
• BVT source parity and no-fixed-sleep scan — PASS
• canonical wrapped-score BVT now covers wrapped-only MATCH on both INNER JOIN
  child positions with qualifying and rejecting thresholds; exact-head remote
  BVT remains pending
• exact-head semantic preflight — PASS semantic=PASS, diff hash
  efc51153938b14e051372026b296c06e05696385ac674fa7f05f3a7217741d09

Scope and acceptance boundaries

• QA required: yes
• The PR was opened as Draft for preflight and is now Ready for review.
  Required CI, formal reviews, and Mergify remain pending; no remote pass is
  claimed until they bind to this exact head.
• Build success, index creation/readiness, and elapsed workload duration are
  not acceptance by themselves. No classic FULLTEXT result, fixed-commit
  product acceptance claim, or formal QPS claim is used here.
• The 2.3M frozen corpus and shared freetier-01 workload remain post-merge
  gates. After merge, only the official immutable image/release workflow may
  produce the tag/digest; freetier-01 must be upgraded through the reviewed
  GitOps environment=new-dev + mode=mo + action=set PR. No local image,
  direct 4.2-dev push, direct GitOps edit, kubectl mutation, or pod restart
  is authorized.
• The first post-release test is a unique-database 15-minute small-corpus
  FULLTEXT2 shakeout with readiness polling, token-bucket DML, C1/C10 readers,
  and an independent correctness ledger. Only MEASURED_PASS with zero
  correctness errors, no backlog, and complete observations may unlock the
  low-load two-hour continuous run.

Please keep issue #25541 open until QA explicitly reports PASS/FAIL.

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[VioletQwQ-0]

@iamlinjunhong Addressed your P1 finding on exact head 772417a0fe3e53420ba46adaeb1c7ffc5c3eba3a.

• applyFullTextFiltersForScanInJoin now collects wrapped FULLTEXT2 MATCHes before the no-filter early return and routes them through applyJoinFullTextIndicesWithWrapped.
• The existing lifting path keeps the score comparison in an executable FILTER above the rewritten JOIN child.
• Added planner coverage for wrapped-only MATCH on both left and right INNER JOIN children, plus canonical BVT coverage for qualifying > 0.0132 and rejecting > 0.9 thresholds.

Validation: focused planner regression PASS, full pkg/sql/plan test suite PASS, git diff --check PASS, and exact-head semantic preflight PASS. The new remote BVT remains pending on this head. Please re-review.

[VioletQwQ-0]

@XuPeng-SH Addressed your P1 finding on exact head 772417a0fe3e53420ba46adaeb1c7ffc5c3eba3a.

• applyFullTextFiltersForScanInJoin now collects wrapped FULLTEXT2 MATCHes before the no-filter early return and routes them through applyJoinFullTextIndicesWithWrapped.
• The existing lifting path keeps the score comparison in an executable FILTER above the rewritten JOIN child.
• Added planner coverage for wrapped-only MATCH on both left and right INNER JOIN children, plus canonical BVT coverage for qualifying > 0.0132 and rejecting > 0.9 thresholds.

Validation: focused planner regression PASS, full pkg/sql/plan test suite PASS, git diff --check PASS, and exact-head semantic preflight PASS. The new remote BVT remains pending on this head. Please re-review.

[VioletQwQ-0]

Follow-up for exact head 325686d29c17b3729f32b837b5bc2144d33f0a9b:

• Fixed the current-head SCA failure from run 33029974462 (job 98380010814): removed the unreferenced applyJoinFullTextIndices wrapper and updated the stale planner comment. All remaining callers use applyJoinFullTextIndicesWithWrapped.
• Local validation passed: the focused wrapped-JOIN-child test, the full pkg/sql/plan test suite, go list, build, vet, and git diff --check.
• Exact-content schema-v6 semantic preflight passed with diff hash efc51153938b14e051372026b296c06e05696385ac674fa7f05f3a7217741d09.
• The new exact-head CI run is pending, including remote SCA and canonical BVT jobs; no remote SCA or BVT pass is claimed yet.

[aunjgr]

Approved at exact head 325686d29c17b3729f32b837b5bc2144d33f0a9b against 4.2-dev@d2393868a7aaa6343518d80849fe4696aa3577e9.

The bounded manifest identifies the fixed main-line behavior closure and the 4.2-specific parser/PERFORM/pipe adaptations. FULLTEXT2 remains separately registered from classic FULLTEXT, with catalog/compile/plan/idxcron/ISCP interface assertions, CPU registration, table-function wiring, CDC/ISCP imports, and public SQL coverage intact. The source/target ownership, cache generation, position-free, persistence/restore, and readiness paths are represented in the backport tests.

The latest JOIN-child fix now collects wrapped-only FULLTEXT2 MATCH drivers before the early return, routes them through the existing served-match rewrite, lifts the score comparison into a post-search FILTER, and keeps upper-bound-only/nonmembership predicates fail-closed. Both INNER JOIN child positions and qualifying/rejecting thresholds are covered at planner and BVT layers. The structural plugin guards found no removed assertions, plugin back-import cycle, or missing registration seam. I found no remaining release-branch correctness or lifecycle blocker.

[XuPeng-SH]

Approved exact head 325686d against 4.2-dev@d2393868a7aaa6343518d80849fe4696aa3577e9. Re-reviewed the complete delta after my prior CHANGES_REQUESTED at a8476d6: wrapped-only FULLTEXT2 MATCH predicates are now collected before the join-child early return, both INNER JOIN child positions route through the existing wrapped rewrite, and the exact score comparison remains in an executable FILTER above the generated search join. The qualifying and rejecting thresholds are covered in planner tests and in the selected fulltext2_wrapped_score BVT. Exact-head UT, SCA, coverage, build, proxy BVT, and pessimistic BVT are green. I found no remaining correctness, liveness, lifecycle, compatibility, or material performance blocker.

[iamlinjunhong]

Re-reviewed exact head 325686d29c17b3729f32b837b5bc2144d33f0a9b against 4.2-dev@d2393868a7aaa6343518d80849fe4696aa3577e9 after my previous CHANGES_REQUESTED.

No findings. The prior P1 is resolved: the JOIN-child path now collects wrapped-only FULLTEXT2 MATCH drivers before the no-filter return, routes both INNER JOIN child positions through applyJoinFullTextIndicesWithWrapped, and evaluates the exact score comparison in a real FILTER above the generated search join. Planner coverage verifies both child positions; the canonical BVT verifies qualifying and rejecting thresholds.

I also re-inspected the complete backport range and its parser/catalog/DDL, plugin registration and interface assertions, table-function execution, CDC/ISCP error propagation, cache-generation and cleanup lifecycle, persistence/restore paths, and focused test coverage. The index-plugin structural searches found no removed assertions, plugin-to-SQL import cycle, or missing CPU/ISCP registration seam. The exact-head pre-merge CI suite was green. The later merge-trigger TKE failure is infrastructure-only (moby/buildkit manifest request returned EOF while the workflow checked out main), not a PR-code failure.