Repository navigation
feat(staged): pause and resume sessions that fail on the network - #1094
Merged
Merged
Conversation
Step 1 of the network pause plan: a pure classifier that answers whether a failed prompt was caused by the network, so a later step can park such a turn and resume it when the network returns. Nothing calls it yet. `classify_network_failure(provider, driver_error, provider_metadata)` lives in `session_runner/network_failure.rs` and returns the signal that matched, for logs and metrics. It reads the structured ACP error stored at `provider_metadata.acp.promptError` and checks, in order: 1. the run failed on the prompt, so a stale `promptError` is ignored; 2. the code is -32603; 3. `data.reason` is not `incoming_transport_closed`; 4. the agent's structured field allows a network cause: `errorKind` of `server_error` or `unknown` for Claude, `reason: turn_failed` for Goose; 5. a row of the fallback table matches for that agent. Only step 5 reads error prose, because no protocol or bridge version marks a failure as network-caused. The table sits in one commented block with the reason for each row: an allow-listed errno token in trailing parentheses, the exact `Request timed out` constant and the HTTP 200 gateway page for Claude, and the `Network error: ` prefix for Goose. TLS certificate codes and `ERR_PROXY_TUNNEL` are left out because they need the user. Codex is never classified. The driver's `Prompt failed:` prefix is now the exported constant `acp_client::PROMPT_FAILED_PREFIX`, used by `describe_prompt_error` and the recorded-turn path. The runner receives that text either bare or wrapped once as the `data` string of an "ACP protocol failed" error, so the prompt-failure check accepts the prefix at the start of the error or at the start of its outermost `data` string, and nowhere deeper. Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Step 2 of the network pause plan: wire the classifier in. A turn that fails because the agent could not reach its model provider is now parked and resumed by the scheduler, instead of ending as `error`/`crashed`. It runs under the existing per-session backoff, eight-attempt cap and 24 hour window; the shared gate that replaces those comes later. The decision for a run that ended in an agent error moves out of `finish_session_run` into `session_runner/failed_run.rs`. It tries, in order: a provider limit, a network failure, then an ordinary failure. The limit check stays first so a limit is never taken for the network. Pipeline sessions are never classified, so a push or pull that fails on the network still ends `error`/`crashed`. A network failure reuses the rate-limit pause: status `paused`, `CompletionReason::RateLimited`, `rate_limit_kind = "network"` and no reset time. No column, migration or type is added. The kind is the new constant `NETWORK_PAUSE_KIND`, and the comment on `CompletionReason::RateLimited` now covers both meanings. Because the pause is the same one, the pause write, the late-Stop handling on both sides of it, the scheduler's claim and the queue gate apply unchanged. An automatic resume of a network pause sends a prompt that tells the agent its turn was cut off and to check its work first. The choice lives in `auto_resume_prompt` and `RateLimitResumeTicket::prompt`; a message the user queued during the pause still replaces it. A network pause records a `network_pause` metric event with the matched signal, in place of a rate-limit hit. Tests cover: a network prompt error parks the row with kind `network` and schedules a retry; a pipeline session with the same error ends `error`/`crashed`; a Stop before or after the pause write ends the row `cancelled`; the auto-resume sends the network prompt; a queued message replaces it; and API-limit pauses keep their kind and prompt. Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Step 3 of the network pause plan, frontend only. A session parked because the agent could not reach its model provider is stored as a `rate_limited` pause with `rateLimitKind` of `network`, so until now the app described it as an API limit. It now says what happened. The timeline row subtitle from `failedArtifactSubtitle` reads "Paused: network unavailable. Resumes when the connection returns." for that kind. The chat alert from `sessionEndMessage` shows the same sentence, and adds "Trying again around 3:45 PM." when the scheduler has a next retry time. For a row the scheduler gave up on or will not resume by itself, the alert reads "Paused: network unavailable. Resume manually when ready.", leaving out the promise to resume because it would be false. The Resume now button is unchanged. API-limit pauses keep their copy. The kind is the constant `NETWORK_PAUSE_KIND` in `sessionFailureCopy.ts`, mirroring the backend constant of the same name. `failedArtifactSubtitle` now takes its optional session fields as one named object instead of four positional arguments. The project note row passed the reset time in the error-message position, so an API-limit pause on a project note never showed its reset time. Named fields fix that call and prevent the mix-up. The three `BranchTimeline.svelte` call sites and the project note row now pass `rateLimitKind`. Tests cover both kinds in the existing helper test files: the subtitle for each artifact, a network pause ignoring a provider reset time, the kind being read only on a rate-limit pause, the alert with and without a retry time, the manual-resume alert, and the project note row for a network pause and a dated API-limit pause. Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Step 4 of the network pause plan: network-paused sessions no longer back off one by one. They wait behind one gate held in memory by the scheduler, which resumes a single session as a probe and the rest once the probe gets through. API-limit pauses never consult it. The gate lives in `rate_limit_scheduler/network_gate.rs`, owned by `RateLimitScheduler` behind a mutex. It starts closed with a probe due at once, so a relaunch probes on its first tick. - Closed: a due tick resumes the oldest due, safe network row as the probe. Probing: no other network resume starts. Open: the rest resume at the existing two per tick. - A probe succeeds when its turn records agent output after the claim, or completes. The new store query `session_has_agent_output_since` backs that check and shares its row filter with `agent_answered_message`. - A probe fails when its session is parked again with kind `network`, or fails to start. The gate closes and the probe delay doubles from 15 seconds to a 10 minute cap. The re-parked row gets a new `updated_at`, so the next probe is a different session. - A probe that was stopped, deleted or ended on another error says nothing about the network, so another row may probe at once. - A failure while the gate is not open is not counted: `attempt` stays, `next_retry_at` is now, the cap and the 24 hour window are skipped, and the bookkeeping reason is `network_wait`. A failure while it is open is counted under the existing rules and closes the gate. `run_tick` settles the probe, then asks `admit_for_resume` for each row: the row's own retry state must be due, and a network row must be let through by the gate. The probe is recorded only after its claim wins, so a deferred or lost claim lets the next network row probe on that tick. `plan_recovery_state` and `build_recovery_state` take the kind and a `counted` flag. `pause_for_rate_limit` takes the scheduler, reports a network failure to the gate and plans the pause from its answer. It settles the probe first, so a probe that answered and then failed is counted. The runner passes the managed scheduler; with none, a network pause is not counted, as under a fresh gate. A failed resume start reports to the gate the same way. Tests drive the synchronous part of the tick with an injected `now`: one probe for eleven paused rows; a failed probe spends no attempt and rotates, with the delay doubling to its cap; agent output or completion opens the gate and the rest resume two per tick; output from before the claim does not; a failure while open advances `attempt` and closes the gate; a row past 24 hours still resumes; API-limit rows keep their schedule and never move the gate; a stopped probe hands over at once; a probe that fails to start closes the gate uncounted; and a fresh scheduler starts closed with an immediate probe. Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Step 5 of the network pause plan: a closed network gate no longer waits out its doubling delay when the machine wakes or its network changes. Both triggers are derived inside the scheduler tick, with no operating-system callbacks. - Wake. The gate records the wall-clock time of each tick. A gap of more than 45 seconds on the 15 second tick means the process was suspended, and the probe delay resets so a probe is due at once. - Settle rule. A probe starts only on a tick whose preceding gap was normal. A short maintenance wake cannot produce two ticks 15 seconds apart, so it never starts a probe; after the lid opens, the second tick does. The first tick of a process has no gap and still probes. - Network change. While the gate is closed and a network-paused row could still resume by itself, each tick reads the machine's interface addresses. A change resets the probe delay. An empty set means no network, and no probe starts; the delay does not advance meanwhile. The reset sets the delay back to 15 seconds and, for a closed gate, makes the next probe due now. A running probe is left to finish. The first read of a closed period is the baseline, not a change, and the baseline is dropped when the gate opens. A failed read clears a "no network" answer, so it cannot block probes. The addresses come through the new `NetworkFingerprint` trait in `network_gate/fingerprint.rs`. The real implementation uses the new `if-addrs` dependency, flagged for review in the plan. It leaves out loopback addresses and, beyond the plan's wording, link-local ones: macOS keeps an `fe80::` address on its `utun` and `awdl` interfaces with no network, so counting them would make the set never empty. `RateLimitScheduler` owns the fingerprint and takes a fake through `with_network_fingerprint`. `run_tick` calls `observe_network_tick` after it settles the probe and before it admits any row. Tests drive the synchronous part of the tick, triggers included, with an injected clock and a fake fingerprint: a long gap resets the probe delay; no probe starts on the tick after a long gap; a gap of exactly 45 seconds is normal; a run of long gaps never starts a probe; a fingerprint change resets the delay; an empty fingerprint blocks the probe without advancing the delay; losing every address blocks a due probe; an unreadable fingerprint does not block; the fingerprint is read only while the gate is closed with network rows; and the real fingerprint holds no loopback or link-local address. Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Step 6 of the network pause plan: a project session that waits on a child now learns that a paused child is not finished and needs nothing from it. Until now the `wait_for_repo_session` description named queued, running and disconnected as the states to wait on again, so a parent that saw `paused` had no guidance and could cancel or restart a child that Staged was about to resume. The description now lists `paused` with those states and says why a child is paused and what happens next: its agent hit a provider limit or lost its network connection, and Staged resumes a paused session by itself, so it needs no follow-up. The wait itself already kept waiting on `paused`, because it returns early only for a terminal state. Two comments now cover both kinds of pause. The `paused` arm of `cancel_repo_session` says the store write drops the completion reason that both kinds carry, so the scheduler never resumes the child and a network pause never probes with it. The status gate in `pipeline_dependency.rs` says a paused dependency resumes by itself for either kind. No behaviour changes in either place. The existing tool-description test now checks the wait-again clause for all four states, the two causes of a pause, and the statement that Staged resumes the session. Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Fixes every finding of the review of b385cb1. The chief one: a failed Claude probe was judged a success, so failures that should have waited for the gate were counted and the gate could open into an outage. Probe success. A failed Claude turn records more than the error result the review named. Stored turns show the CLI's error as an `agent_message_chunk`, a `response_result` with `is_error: true` (its `subtype` still reads `success`), and the same text as the visible assistant row. All three counted as agent output. The probe check is now the store query `session_reached_provider_since`: a tool call or its result, a `response_result` without the error mark, or a `prompt_response`. Message text and plans no longer count, so a text-only probe opens the gate when its turn ends, not at its first chunk. 24 hour window. For a network pause it runs from the first counted failure: `first_limited_at` is set again when `attempt` becomes 1. No field is added. Probe cap. A session whose own probes fail `MAX_CONSECUTIVE_PROBE_FAILURES` (20) times in a row, each with the machine awake and a non-empty fingerprint, gives up with reason `probe_limit_exceeded` and stays paused for a manual resume. A probe during which the process was suspended, or made with an empty or unreadable fingerprint, neither adds to the run nor breaks it. The counts are held in memory by the gate and cleared when it opens. The gate now answers a failure with `PauseAccounting` (`Counted`, `Waiting`, `ProbesExhausted`) in place of a bool. Failed start. A resume that fails to start abandons the probe (another row may probe at once, the delay does not grow) and is counted under the existing cap of eight. Empty fingerprint. It no longer blocks a probe. It only holds back the wake and change triggers, and the doubling schedule keeps running, so a session whose provider is local is probed with no network. Probe timeout. A probe still running after `PROBE_TIMEOUT_MS` (10 minutes) without reaching its provider closes the gate as a failed probe does. The session keeps running and is not counted. Manual resume. `resume_session_for_store` tells the scheduler when its claim wins. A closed gate watches a network-paused session's turn as its probe, so a resume that gets through opens the gate for the rest. Prompt-failure check. The driver reports it as a flag: `SessionSettled::prompt_failed`, read from the transport monitor's existing `answered_error`, carried on the runner's `RunSettlement` to `decide_failed_run`. `failed_on_the_prompt` and its parsing of the SDK's Debug text are gone, and `PROMPT_FAILED_PREFIX` is private to the driver again. Nothing stored changes. A turn reattached to after its connection went has no flag and is not classified. Chat alert. A row whose bookkeeping reason is `network_wait` shows no retry time: its `nextRetryAt` is the time it paused. Row subtitle. A timeline row carries no retry bookkeeping, so it cannot tell a pause that will resume from one that gave up or is unsafe. It now reads "Paused: network unavailable." with no promise. Keeping the promise on healthy rows needs a new timeline payload field, left for review. Tests, each failing on b385cb1: the real row sequence of a failed Claude turn leaves the gate shut and the failure uncounted, also for a tick between those rows and the pause write; a successful result or a prompt response opens the gate and text does not; a counted failure after a 30 hour outage schedules a retry, and the window still ends 24 hours after the first counted failure; twenty failed probes with a network give up, and with none never do; a failed start is counted, hands the probe over and gives up at the cap; an empty fingerprint keeps the doubling schedule and suppresses the wake and change triggers; a hung probe times out; a manual resume opens the gate, or closes it uncounted when it fails; the driver marks an answered prompt error; a run not flagged as a failed prompt is not paused; the alert for a `network_wait` row with a past time; and the subtitle without a promise. The two `cloned_ref_to_slice_refs` warnings are fixed. Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Resolves the two warnings of the review of dddf059. Timed-out probe. A probe that failed on the network only after `PROBE_TIMEOUT_MS` escaped the bound of 20 failed probes: by then the gate was closed, so the failure was answered `Waiting` without touching the session's count, and its row, due at once, was probed again on the next tick. A lone session could repeat that without limit. The gate now remembers each timed-out probe in `timed_out_probes`, with whether the process was suspended during the turn, before or after the timeout. When that session later fails on the network: - the failure is charged to its run of failed probes if the machine stayed awake and the last fingerprint held an address, the conditions of a probe that fails in time, and the twentieth gives up with `probe_limit_exceeded`; - a closed gate waits at least the current probe delay from the failure before its next probe. The delay is not doubled again, because the timeout already doubled it for this probe; - a probe another session is running is left alone. The record is dropped when the session fails or probes again, and all records are cleared when the gate opens, with the failure counts. A late failure with the gate open is counted as before. The comment on `PROBE_TIMEOUT_MS` claimed it was longer than a provider request takes to time out. That was never measured. It now says so, and that a probe may fail after the timeout and is then charged as above. 24 hour window. For a network pause the window is now measured only across counted failures. `first_limited_at` was set again only when `attempt` became 1, so the session that closed the gate (counted once), then waited behind it for more than a day, gave up with `retry_window_elapsed` on its second counted failure. A counted failure now restarts the window when the pause before it was not counted, read from the bookkeeping `reason` that pause left (`network_wait` or `probe_limit_exceeded`). No field is added. The attempt cap is not restarted, and counted failures with no wait between them still give up once they span more than 24 hours. API-limit pauses are unchanged. Tests, each failing on dddf059 except the two noted: twenty late probe failures exhaust a session; a late failure holds the next probe for the probe delay without doubling it, at the gate and through the scheduler for a lone session; a late failure is charged only awake with a network present, and is charged while another session probes without disturbing it; a wait between two counted failures restarts the window. Passing before and after: an opened gate forgets timed-out probes, a late failure never brings a later probe time forward, and waits do not restart the attempt cap. `the_network_window_runs_from_the_first_counted_failure` asserted the old rule and is replaced by `the_network_window_runs_across_counted_failures`. Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A turn that fails because the agent could not reach its model provider (DNS, connection reset, timeout, gateway page) used to end as
error/crashed. It is now parked as a pause and resumed by Staged when the network returns.How it works
Classify.
session_runner/network_failure.rsdecides whether a failed prompt was caused by the network, from the structured ACP error atprovider_metadata.acp.promptError. Only the last step reads error prose, through one commented allow-list per agent (errno tokens,Request timed outand the HTTP 200 gateway page for Claude; theNetwork error:prefix for Goose). TLS certificate and proxy-tunnel errors are left out because they need the user. Codex and pipeline sessions are never classified. The driver now reports a failed prompt as a flag,SessionSettled::prompt_failed.Park. A network failure reuses the rate-limit pause: status
paused,CompletionReason::RateLimited,rate_limit_kind = "network". No column, migration or type is added. The provider-limit check still runs first. The decision for a failed run now lives insession_runner/failed_run.rs.Resume through one gate. Network-paused sessions do not back off one by one. They wait behind a single in-memory gate (
rate_limit_scheduler/network_gate.rs):Wake and network change. Both are derived inside the scheduler tick, with no OS callbacks. A tick gap over 45 seconds means the process was suspended; a change in the machine's interface addresses means the network changed. Either resets the probe delay so a probe is due at once. With no addresses, the wake and change triggers are held back, but the doubling schedule still runs, so a local provider is still probed.
Auto-resume prompt. A resumed network pause tells the agent its turn was cut off and to check its work first. A message the user queued during the pause replaces it.
Project sessions. The
wait_for_repo_sessiondescription now listspausedas a state to wait on again and says Staged resumes it by itself, so a parent does not cancel or restart a paused child.App copy. The timeline row reads "Paused: network unavailable." The chat alert adds "Resumes when the connection returns." with a retry time when one is known, or "Resume manually when ready." when Staged will not resume it.
failedArtifactSubtitlenow takes named fields; this also fixes the project note row, which passed the reset time in the error-message position and so never showed it.For review
if-addrs, used to read interface addresses for the network-change trigger. Loopback and link-local addresses are ignored (macOS keepsfe80::addresses onutun/awdlwith no network).PROBE_TIMEOUT_MS(10 minutes) is not measured against how long a provider request takes to time out; a probe that fails later is handled as described above.Testing
Unit tests cover the classifier, the pause and late-Stop paths, the gate (probe rotation, counting, caps, the 24 hour window, timeouts, manual resume), the wake and network-change triggers with an injected clock and fake fingerprint, the driver flag, and the frontend copy. Pre-push hooks (
staged-ci,crates-test, lint) passed. Not exercised against a live network outage.🤖 Generated with Claude Code