Skip to content

Add Database.releaseMemory and Database.status - #415

Closed
palmoni5 wants to merge 3 commits into
simolus3:mainfrom
palmoni5:database-memory-status
Closed

palmoni5 wants to merge 3 commits into
simolus3:mainfrom
palmoni5:database-memory-status

Conversation

@palmoni5

Copy link
Copy Markdown
Contributor

This adds two native-only methods to Database, next to backup:

  • void releaseMemory() calls sqlite3_db_release_memory, which frees unused pages from the
    connection's page cache. This helps long-lived connections in memory-constrained apps, for
    example after a large scan or when a mobile app goes to the background.
  • ({int current, int highwater}) status(DatabaseStatus option, {bool reset = false}) calls
    sqlite3_db_status. The new DatabaseStatus enum mirrors SQLITE_DBSTATUS_* (values 0 to 13,
    as of SQLite 3.53). Apps can use it to measure how much memory the cache, schema and statements
    of a connection use, and to track cache hit/miss/spill counters.

Details:

  • sqlite3_db_status doesn't set an error message on the connection, so a non-OK result code is
    reported through createExceptionOutsideOfDatabase. In practice this only happens for
    DatabaseStatus.tempBufferSpill on SQLite versions older than 3.51.0 (which is documented on the
    enum value).
  • I used sqlite3_db_status instead of sqlite3_db_status64, since the 64-bit variant also only
    exists since 3.51.0 and would break source: system with older system libraries.

Linux symbols

The Linux builds link with a version script generated from used_symbols.dart (see
hook/build.dart), so only those symbols are exported. The released 3.6.0
libsqlite3.x64.linux.so therefore doesn't export sqlite3_db_status or
sqlite3_db_release_memory.

I added both functions to assets/sqlite3.h and regenerated libsqlite3.g.dart and
used_symbols.dart with tool/generate_bindings.dart. Rerunning the tool gives no further diff.
tool/build_sqlite.dart runs the same build hook, and the compile_sqlite.yml cache key includes
sqlite3/lib/src/hook/compile/**. So the next prebuilt release exports both symbols, and this PR's
CI already tests them on Linux against freshly compiled libraries. Because asset_hashes.dart ties
each package version to the binaries from its own release, a published version with this API
won't download binaries that lack the symbols. No additional change is needed.

Not included

  • sqlite3_interrupt / sqlite3_is_interrupted: covered by feat: Add native bindings for sqlite3_interrupt and sqlite3_is_interrupted #404.
  • sqlite3_release_memory, sqlite3_soft_heap_limit64 and sqlite3_hard_heap_limit64: with the
    prebuilt compile-time options these would be no-ops, so exposing them would be misleading.
    sqlite3_release_memory needs SQLITE_ENABLE_MEMORY_MANAGEMENT, and the heap limits are only
    enforced with memory statistics, which SQLITE_DEFAULT_MEMSTATUS=0 turns off. The db_*
    variants work regardless.
  • WASM: native-only like backup, since the web would need new exports from the WASM build. I can
    add that in a follow-up if you'd like.

Tests

New tests in test/ffi/database_test.dart use a file database with about 400 KB of blobs. They
check that cacheUsed reflects the cache, that reset: true clears cacheHit, that
releaseMemory() shrinks cacheUsed by more than 10x while the connection keeps working, and that
the bundled SQLite accepts every DatabaseStatus value (tagged require_built, so older system
libraries skip it).

  • dart analyze --fatal-infos, dart format --set-exit-if-changed and
    clang-format --style=google on assets/*.h: clean.
  • dart test in sqlite3 (VM, Windows): 195 passed, 5 skipped.
  • dart test -P system with tool/hook_overrides.dart system-os-specific (winsqlite3 3.51.1):
    188 passed, 12 skipped.
  • dart test in sqlite3_connection_pool: 28 passed.
  • dart run run.dart in native_tests (AOT): 180 passed, 5 skipped.

The CHANGELOG entry is under 3.6.1-wip. Since this adds API to Database, feel free to move it to
a minor version.

🤖 Generated with Claude Code

@simolus3 simolus3 left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

I used sqlite3_db_status instead of sqlite3_db_status64, since the 64-bit variant also only exists since 3.51.0 and would break source: system with older system libraries

I would actually prefer using the 64-bit function instead of truncating silently. Binding to a non-existing function only fails when that function is actually called, so this wouldn't break existing apps.

We can document this limitation on status.

WASM: native-only like backup, since the web would need new exports from the WASM build.

I'm not sure we need releaseMemory() on the web, but having the status interface available on all platforms would be nice and I think we should do it in one PR.

Exporting new symbols works by adding them to unstable in wasm_symbols.dart and re-running generate_bindings.dart. But I can also take care of that once the native parts are done.

Comment thread sqlite3/lib/src/ffi/api.dart Outdated
Comment on lines +192 to +193
/// Unlike `sqlite3_release_memory`, this works without SQLite being compiled
/// with `SQLITE_ENABLE_MEMORY_MANAGEMENT`.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that we have no bindings to that function, we shouldn't mention it in this documentation comment.

@palmoni5

Copy link
Copy Markdown
Contributor Author

Thanks for the review! Pushed 3ba0474:

  • sqlite3_db_status64: status now uses the 64-bit function, so values are no longer truncated. The doc comment says it needs SQLite 3.51.0 or newer. The status tests are tagged require_built, so the system preset skips them.
  • releaseMemory docs: removed the mention of sqlite3_release_memory.
  • Web: status and DatabaseStatus moved to CommonDatabase, backed by a new RawSqliteDatabase.sqlite3_db_status64 binding with FFI and WASM implementations. I added sqlite3_db_status64 to unstable in wasm_symbols.dart and re-ran generate_bindings.dart. With an older sqlite3.wasm that lacks the export, status throws an UnsupportedError. releaseMemory is still native-only. The status tests moved to test/common/database.dart, so they run on the web too.

dart analyze --fatal-infos is clean for sqlite3, sqlite3_web and sqlite3_test, and the tests under test/ffi pass. I don't have a wasi sysroot here, so I couldn't build a sqlite3.wasm with the new export or run the browser tests. The web part compiles with dart2js and dart2wasm but hasn't run yet. You offered to take care of the WASM exports, so please take a look at that part.

One thing I noticed: the compile_wasm cache key in compile_sqlite.yml only hashes sqlite3_wasm_build/src/**. A change to tool/wasm_symbols.dart alone might reuse a cached sqlite3.wasm without the new export, which would make the web status tests fail. I didn't touch the workflow.

Binds sqlite3_db_release_memory and sqlite3_db_status on native platforms. Unlike sqlite3_release_memory and the heap limits, both work with the default compile-time options (no SQLITE_ENABLE_MEMORY_MANAGEMENT, SQLITE_DEFAULT_MEMSTATUS=0).
Also document that tempBufferSpill requires SQLite 3.51.0, and that older libraries make Database.status throw for it.
Database.status becomes CommonDatabase.status, backed by a new
RawSqliteDatabase.sqlite3_db_status64 binding on both FFI and WASM.
Using the 64-bit function avoids silently truncating values. It requires
SQLite 3.51.0, which is documented on status(); the binding only fails
when called, so older system libraries keep working otherwise.

On the web, sqlite3_db_status64 is added to the unstable WASM exports.
Older sqlite3.wasm bundles without it throw an UnsupportedError.
releaseMemory stays native-only.

The status tests move to the common database tests so that they also
run on the web.
@palmoni5
palmoni5 force-pushed the database-memory-status branch from 3ba0474 to 569432d Compare September 29, 2026 20:58
@simolus3

Copy link
Copy Markdown
Owner

Thanks for your contributions! I have merged this PR with minor changes in dc7facb.

@simolus3 simolus3 closed this Sep 29, 2026
@palmoni5
palmoni5 deleted the database-memory-status branch September 29, 2026 21:27
@palmoni5

Copy link
Copy Markdown
Contributor Author

Is there an expected release date for a new version? I am eagerly looking forward to it for my app!

@simolus3

Copy link
Copy Markdown
Owner

I have just released version 3.7.0 of the sqlite3 package, which contains your changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants