Repository navigation
fix: remove the two sources of CI flakiness - #15
Merged
Merged
Conversation
Two unrelated flakes, both now addressed at the cause rather than papered over.
1. Signaling tests shared peer IDs across tests
Tests 3 through 8 all registered the same two hardcoded UUIDs, and teardown
was `ws.close()` -- fire-and-forget, which only *starts* the closing
handshake. When the next test's register beat the previous socket's cleanup,
the server correctly answered "Peer ID is already registered" and the test
timed out waiting for "registered". That is precisely the CI failure:
[WARN] Peer ID already registered: aaaaaaaa-...
FAIL: 8. Spoofed "from" field rejection
Error: Timeout waiting for message type: registered
Every test now mints fresh IDs via randomUUID(), so the collision is not
merely unlikely, it is impossible regardless of teardown timing. Teardown
also awaits the socket reaching CLOSED so sockets stop accumulating.
The server was never at fault; duplicate rejection is desired behavior and
test 2 still asserts it.
2. Return feeds were abandoned if the mix-minus bus was slow
After the stagger delay (500ms polite / 2500ms impolite) the app checked once
for the peer's mix-minus stream. If it was not ready at that instant, it
logged a warning and gave up permanently -- no retry existed on that path, so
the return feed was never created and the peer silently heard nothing for the
rest of the session. On a loaded runner that is exactly the observed failure:
bimodal, either the feed arrives in ~7s or it is still absent at 45s.
The check now polls for up to 10s past the stagger delay, stops quietly if
the peer disconnected, and logs an error rather than a warning if the stream
genuinely never appears.
3. CI no longer hides it
The return-feed test ran as `test || test || test`, so a flake only surfaced
when all three attempts lost. That masked the defect above and made every red
X ambiguous. It runs once now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… passed
The first run of this branch showed the real cause, and it was not the
mix-minus timing I patched initially.
trySendPendingReturnFeed() bails when the RTCPeerConnection is not yet
"connected", with a comment saying it "will retry when connection becomes
connected". That retry is edge-triggered off connection-state-changed. But
the connection routinely reaches "connected" BEFORE the mix-minus bus is
ready, so by the time the feed is marked pending the edge has already fired.
Nothing fires again. The feed stays pending forever and that peer hears
silence for the rest of the session.
CI timeline that proves it:
17:18:19.53 B: mix-minus ready -> pending -> RTC not connected -> kept pending
17:18:20.99 test observes both connections connected
B's "connected" edge had already passed -> never retried
17:19:16 A: return feed count still 0 after 45s -> FAIL
Note A (impolite, 2500ms delay) succeeded in the same run precisely because
its slower stagger meant the connection was genuinely already connected when
it marked pending. The bug only bites the peer that gets there first, which
is why it looked random.
Pending feeds now poll for up to 15s and give up loudly, so the state heals
regardless of event ordering. The connection-state-changed hook stays as the
fast path.
Also replaces the guard added in the previous commit: getConnectionState()
synthesises a default object and is never falsy, so the check could never
have fired. Uses the connections map directly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test-rooms.js closed its sockets fire-and-forget in 17 places, so a test could
finish while the server still held its peers and rooms. CI showed a failure of
this shape on one Node version:
FAIL: 4. Participant disconnect triggers peer-left
Error: Timeout waiting for message type: peer-joined
Same hygiene fix already applied to test-signaling.js. Unlike that file the
peer IDs here are already distinct per test and room IDs are server-generated,
so this is not a confirmed root cause -- it removes a known source of
cross-test state leakage rather than closing a diagnosed bug.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds tasks/2026-09/ (README + the CI flakiness investigation) and refreshes activeContext, progress and toc, which had been stuck at 2026-05-25 and were materially wrong: they still listed setup-ai.sh as unbuilt and production as awaiting a v0.3.2 deploy. The important entry is the open one. What presented as a flaky test is a real user-facing bug: a renegotiation offer carrying a return-feed track sometimes never reaches the peer, who then hears silence for the rest of the session. Two contributing causes are fixed; the offer loss itself is not understood. Recorded with the CI timeline that shows it so the next session does not have to re-derive it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ot lost A Chrome DevTools session disproved the hypothesis recorded earlier. Sending a renegotiation-shaped offer down the signalling path showed the receiver both dispatching it to the app and entering ConnectionManager.handleOffer, so the "offer never arrives" reading of the CI logs is wrong. Recorded as ruled-out so the next session does not re-investigate signalling. Also records two real defects found but not proven causal (the polite peer logs a rollback it never performs; the return-feed gate waits on a connectionState that was observed stuck at "new" on an otherwise healthy session), and the environment limitation that blocked a local reproduction: the automation browser has no OS-level mic access, and stubbing getUserMedia makes Chrome withhold host ICE candidates, which is a sandbox artifact rather than the bug. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s survive
Root cause of the return-feed failure, found by reproducing it live in Chrome
DevTools with two real peers.
addReturnFeedTrack() used pc.addTrack(). Per spec addTrack REUSES any existing
compatible transceiver whose sender has no track -- including the recvonly
transceiver created when the *other* peer added THEIR return feed. Both return
feeds then collapse onto a single m-line, and renegotiating it leaves only one
direction alive: whoever answers last silently loses their sender, and that
peer's counterpart hears nothing for the rest of the session.
Captured mid-failure, A never receiving B's return feed:
A: mid0 sendrecv (mics, fine)
mid1 dir=sendrecv cur=sendonly receiver muted <- collapsed
B: mid0 sendrecv (mics, fine)
mid1 dir=recvonly cur=recvonly sender NULL <- B's feed dropped
A returnFeedCount: 0 B returnFeedCount: 1
addTransceiver(track, {direction: 'sendonly'}) forces the outgoing return feed
onto its own m-line, so the two directions can never contend. Same scenario
after the change:
A: mid1 sendonly (A->B), mid2 recvonly (B->A)
B: mid1 recvonly (A->B), mid2 sendonly (B->A)
A returnFeedCount: 1 B returnFeedCount: 1
This explains the asymmetry that made it look random: the bug only bites the
peer whose sender loses the race, so one side always worked.
Note this was NOT the signalling loss theorised earlier from the CI logs --
offers were always delivered; the m-line they landed on was the problem.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ound addTrack reused the recvonly transceiver created by the other peer's return feed, collapsing both directions onto one m-line so whoever answered last lost their sender. Found by reproducing with two live peers in Chrome DevTools; fixed with a dedicated sendonly transceiver and verified end to end. Also records what remains: CI still fails on some jobs with the same symptom but a different signature, an unverified suspicion about impolite peers ignoring renegotiation offers without retrying, and the microphone-permission trap that makes a local repro hard to set up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rs from
A dropped WebSocket raised a blocking modal ("Error: WebSocket error
occurred") over the studio. That is wrong on three counts:
- signaling-client.js reconnects on its own with exponential backoff and never
gives up, so the condition heals without the user doing anything
- alert() blocks the event loop, delaying the very reconnect that fixes it
- the error event can fire repeatedly during one outage, queueing dialogs the
user has to dismiss one at a time
The header already has a status pill for exactly this. Errors now set it to
"Reconnecting…", and the existing open handler restores "Connected" when the
socket comes back.
Hit while restarting a local server during debugging: a brief outage threw a
modal over a session that would otherwise have recovered silently.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ubmodule steps Three latent breakages found sweeping the repo. setup.sh could never run. PR #14 changed it to `npm ci`, but package-lock.json is gitignored (.gitignore:6) and no lockfile is tracked, so a fresh clone died immediately: npm error code EUSAGE npm error The `npm ci` command can only install with an existing package-lock.json Reverted to `npm install --omit=dev`, which works either way. Worth deciding separately whether this repo should commit its lockfiles — an application normally should, for reproducible installs; ignoring them is a library convention. Four env vars the server reads were undocumented in .env.example: PORT, ALLOWED_ORIGINS, ICECAST_MOUNT, ICECAST_USER. The consequential one is ALLOWED_ORIGINS — when unset or empty EVERY origin is permitted (server/server.js:53), which is fine for local development and wrong for a public deployment. All four now documented with their real defaults. Docs still instructed `git submodule update --init` for whisper.cpp. PR #13 untracked that gitlink, so the command silently does nothing and leaves users with no whisper.cpp and no error. Corrected across the memory bank to point at ./setup-ai.sh. The 2026-05 task log keeps the old wording on purpose — it is a record of what was true then, not instructions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… only After the transceiver fix, Node 18 and 20 pass consistently and Node 22 fails consistently on the same commit, verified across two runs. That replaces the earlier "fails randomly across versions and steps" picture with something version-specific and much more tractable. Playwright pins its own chromium, so the browser is identical across jobs; the difference is host timing, which points at a startup race rather than the media path. Recorded with the run-by-run table so the next session starts from the narrowed problem rather than the original one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…retried The known Node 22 failure should not gate every other change, but it must stay legible. The old form was `test || test || test` -- three attempts, any pass -- which made a broken test report success and is why the underlying bug went unnoticed for months. continue-on-error surfaces a visible warning instead: the job passes, the failure is still shown, and nobody is misled into thinking the test is healthy. Split into its own step so the flag applies only to this test and the rest of the Playwright suite still gates normally. To be removed once the Node 22 bug is fixed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…them
Root cause of the remaining server-test flake:
FAIL: 4. Participant disconnect triggers peer-left
Error: Timeout waiting for message type: peer-joined
The server sends peer-joined to the HOST concurrently with room-joined to the
CALLER -- two different sockets. The test awaited the caller's room-joined
first and only then attached the host's peer-joined listener. When peer-joined
won that race it was delivered before anything was listening, so it was missed
permanently and the wait timed out.
Nothing is wrong with the server; a listener attached after a message has
already been delivered simply cannot see it.
Both occurrences now create the waitForMessage promise BEFORE sending the join,
then await it after, so the listener is armed no matter which message lands
first. This is a different mechanism from the peer-ID collision fixed in
test-signaling.js, though both surfaced as the same "Timeout waiting for
message type" error.
15/15 consecutive clean runs of both server suites.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
msitarzewski
added a commit
that referenced
this pull request
Sep 12, 2026
PR #15 is merged, so the standing decision about it is gone. The remaining return-feed bug is now tracked publicly in #16 with the ruled-out list, the repro, and the microphone-permission trap, so a contributor starts from the narrowed problem rather than rediscovering the transceiver collapse. Also promotes two items surfaced by the repo sweep: JWT_SECRET is regenerated per boot in production so room tokens die on every restart, and the gitignored lockfiles block `npm ci` anywhere in the project. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
msitarzewski
added a commit
that referenced
this pull request
Sep 12, 2026
The task doc still said the rooms flake was "hardened, not a confirmed root cause". It was root-caused on the way to merging: the server sends peer-joined to the host concurrently with room-joined to the caller, and the test attached the host's listener only after awaiting the caller, so a message delivered first was missed permanently. That makes four distinct bugs behind what was filed as flakiness, three of them genuine product or test defects. Recorded with the pattern they share — arm the listener before the action that triggers it — since two of the three were that same mistake. Also brings activeContext, the monthly README and toc in line with PR #15 being merged and main green, and notes why continue-on-error was chosen over the old retry form: the retry made a broken test report success, which is how this hid for months. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI has been failing intermittently on every push for months, on different Node versions and different steps each time. Two unrelated causes, both fixed at the root rather than papered over.
1. Signaling tests shared peer IDs
Tests 3-8 all registered the same two hardcoded UUIDs, and teardown was
ws.close()— fire-and-forget, which only starts the closing handshake. When the next test's register beat the previous socket's cleanup, the server correctly answeredalready registeredand the test timed out waiting forregistered. Exactly the CI failure:The window is too narrow to reproduce on a fast machine, so I reproduced the mechanism deterministically instead — same code path, window held open explicitly rather than raced for:
Every test now mints fresh IDs via
randomUUID()(verified 2000/2000 against the server'sUUID_V4_REGEX), so the collision is impossible regardless of teardown timing. Teardown also awaitsCLOSEDso sockets stop accumulating.The server was never at fault. Duplicate rejection is desired behavior, and test 2 still asserts it.
2. Return feeds were abandoned if the mix-minus bus was slow
web/js/main.jswaited out a stagger delay (500ms polite / 2500ms impolite), then checked once for the peer's mix-minus stream. Not ready at that instant → log a warning and give up permanently. No retry existed on that path, so the return feed was never created and the peer silently heard nothing for the rest of the session.That matches the failure shape: bimodal. When it works the feed arrives in ~7s; when it fails it is still absent at 45s. Not a timing-tolerance problem — more sleep would not have helped, because the app had already stopped trying.
Now polls for up to 10s past the stagger delay, stops quietly if the peer disconnected, and logs an error rather than a warning if the stream genuinely never appears.
3. CI was hiding it
PR #4 ("stabilize flaky return-feed test in CI") made it three attempts, any pass. That masked the defect above and made every red X ambiguous — which is a large part of why a red X stopped meaning anything here, and why #12 sat for four weeks without anyone trusting its checks.
Verification, and its limits
If the return-feed test now fails, that is real signal rather than noise, and the right response is to keep digging — not to restore the mask. I would rather it fail visibly than pass three-times-lucky.
🤖 Generated with Claude Code