Skip to content

Restore SQLite transaction serialization with separate CDC snapshot readers - #650

Merged
wolfy-j merged 1 commit into
mainfrom
fix/sqlite-concurrency
Sep 6, 2026
Merged

wolfy-j merged 1 commit into
mainfrom
fix/sqlite-concurrency

Conversation

@wolfy-j

@wolfy-j wolfy-j commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

PR #351 widened file-backed SQLite application pools from one connection to the configured width (unlimited by default). This changed transaction semantics even without CDC: deferred transactions could read a WAL snapshot, race another writer, and fail when upgrading to a write. The rqlite driver already has a five-second busy timeout; that does not resolve SQLITE_BUSY_SNAPSHOT.

Restore one application connection, held for the whole transaction, as before #351. Give file-backed CDC snapshots a separate, lazily opened read-only pool with one connection. Its connector belongs to the same pool generation, carries no mutation hooks, and closes with the observer. Snapshot admission acquires the reader before the commit fence, preserving snapshot-to-live ordering without consuming the application writer. Private in-memory databases retain their existing single-connection path.

Validation:

  • Added a no-CDC transaction regression: a competing writer must wait until the read/write transaction commits; both updates remain visible. It fails on the previous implementation.
  • Added read-only/lazy snapshot-pool, cancellation, and writer-progress checks. Existing concurrent snapshot-to-live and in-flight-writer coverage passes with the restored application limit.
  • SQL suites pass both without hooks and with sqlite_preupdate_hook under the race detector; SQLite CDC suite and full lint pass.
  • Booted Hub kickside/kickside@0.1.131 with fresh databases. The baseline failed in Keeper reconciliation with Edge insert failed: database is locked. The fix completes all 8 bootloaders and 96 reconciliation batches (9,515 commands, 2,956 entries, zero failures); the homepage responds HTTP 200.

The existing overflow test now waits for asynchronous overflow processing before receiving from its stream; receiving immediately could itself free capacity and invalidate the test's expectation.

@wolfy-j
wolfy-j requested a review from skhaz September 5, 2026 21:21
@wolfy-j
wolfy-j merged commit 20a7c19 into main Sep 6, 2026
8 checks passed
@wolfy-j
wolfy-j deleted the fix/sqlite-concurrency branch September 6, 2026 23:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants