Skip to content

feat(embedding): add native C ABI and lifecycle foundation - #14

Merged
aunjgr merged 2 commits into
matrixorigin:upstream-dev-mergefrom
aunjgr:feature/28966-native-abi
Sep 16, 2026
Merged

aunjgr merged 2 commits into
matrixorigin:upstream-dev-mergefrom
aunjgr:feature/28966-native-abi

Conversation

@aunjgr

@aunjgr aunjgr commented Sep 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Refs matrixorigin/matrixone#28966, migration stage 3 (native foundation).
Based on merged Sirius #13 (93df6a80).

This PR completes the stage 3 control/library foundation needed to embed Sirius
without giving foreign callers ownership of DuckDB's thread-affine execution
window. It is independently reviewable; the four-stage native round is not yet
complete. This branch advertises ENGINE_CONTROL only. The production query factory intentionally returns
UNSUPPORTED until stages 4-6 connect bounded MO input, TAE bindings and results.
It does not call the eager Arrow/materialized-result FFI as a substitute.

Implemented

  • C99 ABI version 1 with opaque handles, bounded caller-owned errors, explicit
    option layout checks and non-clobbering create/close ownership.
  • Static Sirius::embed target and a verified build-tree CMake SDK/linker
    manifest generated from the actual successful C consumer link command.
  • Single native coordinator owns initialization, prepare/run/finish and
    destruction. Cancellation bypasses its command queue; live handles and
    waiting requests are bounded. Wait timeout is distinct from query deadline.
  • Execution that returns normally after its deadline reports TIMEOUT and still
    completes cleanup. SDK export reruns when its generator or public header changes.
  • Shared-lifetime strict byte/count credits retained through lease movement
    and asynchronous ownership, without an oversized-request escape hatch.
  • Native GPU worker default of two, configurable 1-128. Explicit initialization
    disables automatic extension loading to avoid a duplicate Sirius runtime.
  • Fatal synchronization is recorded independently of the root completion
    promise. Workers notify; the query owner seals health and drains. Unprovable
    native cleanup retains unsafe owners and refuses reuse/destruction.

See docs/embedding-c-api.md for API behavior, current limitations and build commands.

Validation

  • Pixi release build of the native library and real C99 consumer passed.
  • Native control/credit suite: 71 assertions in 11 cases passed.
  • The same suite passed ThreadSanitizer.
  • Existing affected GPU pipeline, completion, C++ FFI and TAE suites:
    313 assertions in 21 cases passed.
  • C consumer passed actual GPU runtime creation, unsupported-query cleanup,
    cancellation, duplicate-runtime rejection, and shutdown/recreation with the
    default two GPU workers followed by one worker.
  • A separate CMake project found the generated SDK, linked the C consumer,
    and passed its actual GPU runtime smoke test without access to internal
    DuckDB/cuDF/C++ headers.
  • Focused clang-format, Python syntax and orphan-test registration checks passed.
  • No container builds. No SF10, MO-reader or native-result execution claim.

Follow-up PRs (outside this foundation PR)

  • Stage 4: bounded MO-native source, pooled staging and GPU conversion.
  • Stage 5: query-local Substrait/TAE bindings and bounded TAE preparation.
  • Stage 6: incremental native result packing and scheduling-aware output pressure.
  • End-to-end C producer/result consumer with two GPU streams, cancellation and
    capacity checks across the actual data path.

MO CGo wiring, numeric compatibility and Flight retirement remain outside this round.

@aunjgr
aunjgr marked this pull request as ready for review September 16, 2026 08:46
@aunjgr
aunjgr merged commit f67d9e7 into matrixorigin:upstream-dev-merge Sep 16, 2026
3 of 7 checks passed
zuyu pushed a commit that referenced this pull request Oct 14, 2026
* feat(embedding): add native C ABI and lifecycle foundation

* fix(embedding): enforce execution deadlines and complete foundation validation
zuyu pushed a commit that referenced this pull request Oct 21, 2026
* feat(embedding): add native C ABI and lifecycle foundation

* fix(embedding): enforce execution deadlines and complete foundation validation
zuyu pushed a commit that referenced this pull request Oct 22, 2026
* feat(embedding): add native C ABI and lifecycle foundation

* fix(embedding): enforce execution deadlines and complete foundation validation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant