Skip to content

Keep an h1 upgraded stream open when the 101 is read before the request end - #1051

Open
mikef-eng wants to merge 2 commits into
cloudflare:mainfrom
chelabase:chelabase/websocket-upgrade-race-main
Open

mikef-eng wants to merge 2 commits into
cloudflare:mainfrom
chelabase:chelabase/websocket-upgrade-race-main

Conversation

@mikef-eng

Copy link
Copy Markdown

The race

For an HTTP/1.1 websocket upgrade (a GET without a body), proxy_handle_downstream queues HttpTask::Body(None, true) for the upstream half before the duplex loop starts. If the upstream's 101 is already readable when proxy_handle_upstream first polls, select! can read the 101 first. The queued pre-upgrade end of body is then handled after the upgrade, and request_done && client_session.was_upgraded() takes it for the end of the upgraded stream. The loop exits and both connections close right after the 101. When the end of body happens to be handled first, the existing request_done = false reset on upgrade keeps the connection open, so the failure is intermittent.

The fix

After the 101, the downstream half sends only UpgradedBody tasks, so a plain Body end handled after the upgrade can only be the request's own pre-upgrade end. In that case proxy_handle_upstream now keeps receiving the upgraded body (request_done = false), the same as when that end is handled before the 101. An UpgradedBody end or a dropped sender still ends the upgraded stream.

The test

proxy_h1::tests::upgraded_stream_survives_request_end_handled_after_101 drives proxy_handle_upstream over a tokio::io::duplex stream and forces the losing order: the peer writes the 101, the test waits for the Header task, and only then sends Body(None, true) and UpgradedBody(Some("ping"), false). It asserts that ping reaches the upstream and that UpgradedBody(None, true) still ends the loop. Without the fix it fails, because ping is never written. With the fix, cargo test -p pingora-proxy --lib passes (59 tests).

Fixes #1050

🤖 Generated with Claude Code

mikef-eng and others added 2 commits October 15, 2026 07:11
…st end

For a request without a body (a websocket GET), the downstream half queues
`Body(None, true)` for the upstream half before the duplex loop starts. If the
upstream's 101 is already readable when `proxy_handle_upstream` first polls,
`select!` may read the 101 first. The queued pre-upgrade end of body is then
handled after the upgrade, and `request_done && was_upgraded()` takes it for
the end of the upgraded stream: the loop exits and both connections close right
after the 101.

A plain `Body` end handled after the upgrade now keeps receiving the upgraded
body, as when it is handled before the 101. Only an `UpgradedBody` end or a
dropped sender ends the upgraded stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015aL1hx3MokwQ9Zw5tvJcAb
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015aL1hx3MokwQ9Zw5tvJcAb
@dorianverlaine

Copy link
Copy Markdown

Same race as #947 (open since Aug 4) — good to see it described this cleanly.

We have been running the fix from #947 in production since August (backported onto pingora 0.9.0), and the issue has picked up a few independent confirmations since. Two details from that PR that might be worth folding in here if maintainers want this one to be the landing:

  • it treats an explicitly dropped downstream pipe (body.is_none()) as the end of the tunnel;
  • its coverage is integration-level (pingora-proxy/tests/test_upstream.rs, real client and upstream connections).

Happy to close #947 in favor of this PR and port either of those over — either way, good to see this race getting a clean fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HTTP/1.1 websocket upgrade closed right after the 101 when the 101 is read before the request's end of body

2 participants