Repository navigation
Fix Mentions Watcher OOM and Tap Sync at_uri conflicts - #56
Open
hipstersmoothie wants to merge 1 commit into
Open
hipstersmoothie wants to merge 1 commit into
hipstersmoothie wants to merge 1 commit into
Conversation
Jetstream consumer: process messages serially through a bounded queue, pausing the socket above 1000 queued messages, and throttle cursor saves to once per second. Previously every message spawned a concurrent DB write, so replaying a saved cursor piled up handlers until the 4 GB heap ran out, leaving the service in a crash loop. Tap listing ingest: reconcile the record onto the row that owns its at_uri before the slug-keyed upsert. A record whose slug didn't match its existing row failed on store_listings_at_uri_idx and was retried forever. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prod issues
Mentions Watcher crash loop (OOM). The service dies roughly 220s after each start with
JavaScript heap out of memoryat 4 GB. Each Jetstream message (about 3.5k/sec while catching up) started a handler that wrote the cursor to the DB, with no backpressure. After 181s there were 621k messages but only 1,047 cursor saves, so pending handlers piled up. Each restart resumes further behind, so the service can't recover on its own.Tap Sync retrying events forever. Events 5620 and 6419 fail on
store_listings_at_uri_idx.upsertDirectoryListingFromTapmatches rows byslug, butat_uriis also unique. When a record's slug doesn't match the row that already holds itsat_uri, the upsert fails.resolveTapIngestSluglooks up the row byat_urifirst. If the record's slug is free, the row takes it. If another listing already uses it, the row keeps its current slug and a warning is logged.Testing
tsc --noEmit, oxlint and vitest (108 tests) pass. No test covers the new code paths.Not fixed here
ATSTORE_IDENTIFIERandATSTORE_APP_PASSWORDset in Railway. Page snapshot refresh fails for nearly every listing without them.🤖 Generated with Claude Code