Skip to content

feat(staged): pause and resume sessions that fail on the network - #1094

Merged
matt2e merged 8 commits into
mainfrom
retry-on-network-or-dns-error
Nov 20, 2026
Merged

matt2e merged 8 commits into
mainfrom
retry-on-network-or-dns-error

Conversation

@matt2e

@matt2e matt2e commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Summary

A turn that fails because the agent could not reach its model provider (DNS, connection reset, timeout, gateway page) used to end as error/crashed. It is now parked as a pause and resumed by Staged when the network returns.

How it works

Classify. session_runner/network_failure.rs decides whether a failed prompt was caused by the network, from the structured ACP error at provider_metadata.acp.promptError. Only the last step reads error prose, through one commented allow-list per agent (errno tokens, Request timed out and the HTTP 200 gateway page for Claude; the Network error: prefix for Goose). TLS certificate and proxy-tunnel errors are left out because they need the user. Codex and pipeline sessions are never classified. The driver now reports a failed prompt as a flag, SessionSettled::prompt_failed.

Park. A network failure reuses the rate-limit pause: status paused, CompletionReason::RateLimited, rate_limit_kind = "network". No column, migration or type is added. The provider-limit check still runs first. The decision for a failed run now lives in session_runner/failed_run.rs.

Resume through one gate. Network-paused sessions do not back off one by one. They wait behind a single in-memory gate (rate_limit_scheduler/network_gate.rs):

  • Closed: one session is resumed as a probe. The probe delay doubles from 15 seconds to a 10 minute cap on each failure.
  • A probe succeeds once its session reaches the provider (a tool call or result, a non-error result, or a prompt response). The gate then opens and the rest resume two per tick.
  • A failure behind a closed gate is not counted against the session's eight-attempt cap or 24 hour window. A failure with the gate open is counted and closes it.
  • A session whose own probes fail 20 times in a row with the machine awake and a network present gives up and waits for a manual resume. Probes that fail only after the 10 minute probe timeout count toward that limit too.
  • A manual resume of a network-paused session acts as the probe.
  • API-limit pauses never consult the gate and keep their schedule.

Wake and network change. Both are derived inside the scheduler tick, with no OS callbacks. A tick gap over 45 seconds means the process was suspended; a change in the machine's interface addresses means the network changed. Either resets the probe delay so a probe is due at once. With no addresses, the wake and change triggers are held back, but the doubling schedule still runs, so a local provider is still probed.

Auto-resume prompt. A resumed network pause tells the agent its turn was cut off and to check its work first. A message the user queued during the pause replaces it.

Project sessions. The wait_for_repo_session description now lists paused as a state to wait on again and says Staged resumes it by itself, so a parent does not cancel or restart a paused child.

App copy. The timeline row reads "Paused: network unavailable." The chat alert adds "Resumes when the connection returns." with a retry time when one is known, or "Resume manually when ready." when Staged will not resume it. failedArtifactSubtitle now takes named fields; this also fixes the project note row, which passed the reset time in the error-message position and so never showed it.

For review

  • New dependency: if-addrs, used to read interface addresses for the network-change trigger. Loopback and link-local addresses are ignored (macOS keeps fe80:: addresses on utun/awdl with no network).
  • No data model changes. The network pause reuses existing fields. One consequence: a timeline row carries no retry bookkeeping, so its subtitle makes no promise to resume. Restoring that promise on healthy rows needs a new timeline payload field, which is left out pending review.
  • PROBE_TIMEOUT_MS (10 minutes) is not measured against how long a provider request takes to time out; a probe that fails later is handled as described above.

Testing

Unit tests cover the classifier, the pause and late-Stop paths, the gate (probe rotation, counting, caps, the 24 hour window, timeouts, manual resume), the wake and network-change triggers with an injected clock and fake fingerprint, the driver flag, and the frontend copy. Pre-push hooks (staged-ci, crates-test, lint) passed. Not exercised against a live network outage.

🤖 Generated with Claude Code

matt2e and others added 8 commits November 19, 2026 14:54
Step 1 of the network pause plan: a pure classifier that answers whether
a failed prompt was caused by the network, so a later step can park such
a turn and resume it when the network returns. Nothing calls it yet.

`classify_network_failure(provider, driver_error, provider_metadata)`
lives in `session_runner/network_failure.rs` and returns the signal that
matched, for logs and metrics. It reads the structured ACP error stored
at `provider_metadata.acp.promptError` and checks, in order:

1. the run failed on the prompt, so a stale `promptError` is ignored;
2. the code is -32603;
3. `data.reason` is not `incoming_transport_closed`;
4. the agent's structured field allows a network cause: `errorKind` of
   `server_error` or `unknown` for Claude, `reason: turn_failed` for
   Goose;
5. a row of the fallback table matches for that agent.

Only step 5 reads error prose, because no protocol or bridge version
marks a failure as network-caused. The table sits in one commented block
with the reason for each row: an allow-listed errno token in trailing
parentheses, the exact `Request timed out` constant and the HTTP 200
gateway page for Claude, and the `Network error: ` prefix for Goose. TLS
certificate codes and `ERR_PROXY_TUNNEL` are left out because they need
the user. Codex is never classified.

The driver's `Prompt failed:` prefix is now the exported constant
`acp_client::PROMPT_FAILED_PREFIX`, used by `describe_prompt_error` and
the recorded-turn path. The runner receives that text either bare or
wrapped once as the `data` string of an "ACP protocol failed" error, so
the prompt-failure check accepts the prefix at the start of the error or
at the start of its outermost `data` string, and nowhere deeper.

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Step 2 of the network pause plan: wire the classifier in. A turn that
fails because the agent could not reach its model provider is now parked
and resumed by the scheduler, instead of ending as `error`/`crashed`.
It runs under the existing per-session backoff, eight-attempt cap and
24 hour window; the shared gate that replaces those comes later.

The decision for a run that ended in an agent error moves out of
`finish_session_run` into `session_runner/failed_run.rs`. It tries, in
order: a provider limit, a network failure, then an ordinary failure.
The limit check stays first so a limit is never taken for the network.
Pipeline sessions are never classified, so a push or pull that fails on
the network still ends `error`/`crashed`.

A network failure reuses the rate-limit pause: status `paused`,
`CompletionReason::RateLimited`, `rate_limit_kind = "network"` and no
reset time. No column, migration or type is added. The kind is the new
constant `NETWORK_PAUSE_KIND`, and the comment on
`CompletionReason::RateLimited` now covers both meanings. Because the
pause is the same one, the pause write, the late-Stop handling on both
sides of it, the scheduler's claim and the queue gate apply unchanged.

An automatic resume of a network pause sends a prompt that tells the
agent its turn was cut off and to check its work first. The choice lives
in `auto_resume_prompt` and `RateLimitResumeTicket::prompt`; a message
the user queued during the pause still replaces it.

A network pause records a `network_pause` metric event with the matched
signal, in place of a rate-limit hit.

Tests cover: a network prompt error parks the row with kind `network`
and schedules a retry; a pipeline session with the same error ends
`error`/`crashed`; a Stop before or after the pause write ends the row
`cancelled`; the auto-resume sends the network prompt; a queued message
replaces it; and API-limit pauses keep their kind and prompt.

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Step 3 of the network pause plan, frontend only. A session parked
because the agent could not reach its model provider is stored as a
`rate_limited` pause with `rateLimitKind` of `network`, so until now the
app described it as an API limit. It now says what happened.

The timeline row subtitle from `failedArtifactSubtitle` reads "Paused:
network unavailable. Resumes when the connection returns." for that
kind. The chat alert from `sessionEndMessage` shows the same sentence,
and adds "Trying again around 3:45 PM." when the scheduler has a next
retry time. For a row the scheduler gave up on or will not resume by
itself, the alert reads "Paused: network unavailable. Resume manually
when ready.", leaving out the promise to resume because it would be
false. The Resume now button is unchanged. API-limit pauses keep their
copy.

The kind is the constant `NETWORK_PAUSE_KIND` in `sessionFailureCopy.ts`,
mirroring the backend constant of the same name.

`failedArtifactSubtitle` now takes its optional session fields as one
named object instead of four positional arguments. The project note row
passed the reset time in the error-message position, so an API-limit
pause on a project note never showed its reset time. Named fields fix
that call and prevent the mix-up. The three `BranchTimeline.svelte` call
sites and the project note row now pass `rateLimitKind`.

Tests cover both kinds in the existing helper test files: the subtitle
for each artifact, a network pause ignoring a provider reset time, the
kind being read only on a rate-limit pause, the alert with and without
a retry time, the manual-resume alert, and the project note row for a
network pause and a dated API-limit pause.

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Step 4 of the network pause plan: network-paused sessions no longer back
off one by one. They wait behind one gate held in memory by the
scheduler, which resumes a single session as a probe and the rest once
the probe gets through. API-limit pauses never consult it.

The gate lives in `rate_limit_scheduler/network_gate.rs`, owned by
`RateLimitScheduler` behind a mutex. It starts closed with a probe due
at once, so a relaunch probes on its first tick.

- Closed: a due tick resumes the oldest due, safe network row as the
  probe. Probing: no other network resume starts. Open: the rest resume
  at the existing two per tick.
- A probe succeeds when its turn records agent output after the claim,
  or completes. The new store query `session_has_agent_output_since`
  backs that check and shares its row filter with
  `agent_answered_message`.
- A probe fails when its session is parked again with kind `network`,
  or fails to start. The gate closes and the probe delay doubles from
  15 seconds to a 10 minute cap. The re-parked row gets a new
  `updated_at`, so the next probe is a different session.
- A probe that was stopped, deleted or ended on another error says
  nothing about the network, so another row may probe at once.
- A failure while the gate is not open is not counted: `attempt` stays,
  `next_retry_at` is now, the cap and the 24 hour window are skipped,
  and the bookkeeping reason is `network_wait`. A failure while it is
  open is counted under the existing rules and closes the gate.

`run_tick` settles the probe, then asks `admit_for_resume` for each row:
the row's own retry state must be due, and a network row must be let
through by the gate. The probe is recorded only after its claim wins, so
a deferred or lost claim lets the next network row probe on that tick.

`plan_recovery_state` and `build_recovery_state` take the kind and a
`counted` flag. `pause_for_rate_limit` takes the scheduler, reports a
network failure to the gate and plans the pause from its answer. It
settles the probe first, so a probe that answered and then failed is
counted. The runner passes the managed scheduler; with none, a network
pause is not counted, as under a fresh gate. A failed resume start
reports to the gate the same way.

Tests drive the synchronous part of the tick with an injected `now`:
one probe for eleven paused rows; a failed probe spends no attempt and
rotates, with the delay doubling to its cap; agent output or completion
opens the gate and the rest resume two per tick; output from before the
claim does not; a failure while open advances `attempt` and closes the
gate; a row past 24 hours still resumes; API-limit rows keep their
schedule and never move the gate; a stopped probe hands over at once; a
probe that fails to start closes the gate uncounted; and a fresh
scheduler starts closed with an immediate probe.

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Step 5 of the network pause plan: a closed network gate no longer waits
out its doubling delay when the machine wakes or its network changes.
Both triggers are derived inside the scheduler tick, with no
operating-system callbacks.

- Wake. The gate records the wall-clock time of each tick. A gap of
  more than 45 seconds on the 15 second tick means the process was
  suspended, and the probe delay resets so a probe is due at once.
- Settle rule. A probe starts only on a tick whose preceding gap was
  normal. A short maintenance wake cannot produce two ticks 15 seconds
  apart, so it never starts a probe; after the lid opens, the second
  tick does. The first tick of a process has no gap and still probes.
- Network change. While the gate is closed and a network-paused row
  could still resume by itself, each tick reads the machine's interface
  addresses. A change resets the probe delay. An empty set means no
  network, and no probe starts; the delay does not advance meanwhile.

The reset sets the delay back to 15 seconds and, for a closed gate,
makes the next probe due now. A running probe is left to finish. The
first read of a closed period is the baseline, not a change, and the
baseline is dropped when the gate opens. A failed read clears a "no
network" answer, so it cannot block probes.

The addresses come through the new `NetworkFingerprint` trait in
`network_gate/fingerprint.rs`. The real implementation uses the new
`if-addrs` dependency, flagged for review in the plan. It leaves out
loopback addresses and, beyond the plan's wording, link-local ones:
macOS keeps an `fe80::` address on its `utun` and `awdl` interfaces
with no network, so counting them would make the set never empty.
`RateLimitScheduler` owns the fingerprint and takes a fake through
`with_network_fingerprint`.

`run_tick` calls `observe_network_tick` after it settles the probe and
before it admits any row.

Tests drive the synchronous part of the tick, triggers included, with
an injected clock and a fake fingerprint: a long gap resets the probe
delay; no probe starts on the tick after a long gap; a gap of exactly
45 seconds is normal; a run of long gaps never starts a probe; a
fingerprint change resets the delay; an empty fingerprint blocks the
probe without advancing the delay; losing every address blocks a due
probe; an unreadable fingerprint does not block; the fingerprint is
read only while the gate is closed with network rows; and the real
fingerprint holds no loopback or link-local address.

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Step 6 of the network pause plan: a project session that waits on a
child now learns that a paused child is not finished and needs nothing
from it. Until now the `wait_for_repo_session` description named
queued, running and disconnected as the states to wait on again, so a
parent that saw `paused` had no guidance and could cancel or restart a
child that Staged was about to resume.

The description now lists `paused` with those states and says why a
child is paused and what happens next: its agent hit a provider limit
or lost its network connection, and Staged resumes a paused session by
itself, so it needs no follow-up. The wait itself already kept waiting
on `paused`, because it returns early only for a terminal state.

Two comments now cover both kinds of pause. The `paused` arm of
`cancel_repo_session` says the store write drops the completion reason
that both kinds carry, so the scheduler never resumes the child and a
network pause never probes with it. The status gate in
`pipeline_dependency.rs` says a paused dependency resumes by itself for
either kind. No behaviour changes in either place.

The existing tool-description test now checks the wait-again clause for
all four states, the two causes of a pause, and the statement that
Staged resumes the session.

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Fixes every finding of the review of b385cb1. The chief one: a failed
Claude probe was judged a success, so failures that should have waited
for the gate were counted and the gate could open into an outage.

Probe success. A failed Claude turn records more than the error result
the review named. Stored turns show the CLI's error as an
`agent_message_chunk`, a `response_result` with `is_error: true` (its
`subtype` still reads `success`), and the same text as the visible
assistant row. All three counted as agent output. The probe check is now
the store query `session_reached_provider_since`: a tool call or its
result, a `response_result` without the error mark, or a
`prompt_response`. Message text and plans no longer count, so a
text-only probe opens the gate when its turn ends, not at its first
chunk.

24 hour window. For a network pause it runs from the first counted
failure: `first_limited_at` is set again when `attempt` becomes 1. No
field is added.

Probe cap. A session whose own probes fail
`MAX_CONSECUTIVE_PROBE_FAILURES` (20) times in a row, each with the
machine awake and a non-empty fingerprint, gives up with reason
`probe_limit_exceeded` and stays paused for a manual resume. A probe
during which the process was suspended, or made with an empty or
unreadable fingerprint, neither adds to the run nor breaks it. The
counts are held in memory by the gate and cleared when it opens.

The gate now answers a failure with `PauseAccounting` (`Counted`,
`Waiting`, `ProbesExhausted`) in place of a bool.

Failed start. A resume that fails to start abandons the probe (another
row may probe at once, the delay does not grow) and is counted under the
existing cap of eight.

Empty fingerprint. It no longer blocks a probe. It only holds back the
wake and change triggers, and the doubling schedule keeps running, so a
session whose provider is local is probed with no network.

Probe timeout. A probe still running after `PROBE_TIMEOUT_MS` (10
minutes) without reaching its provider closes the gate as a failed probe
does. The session keeps running and is not counted.

Manual resume. `resume_session_for_store` tells the scheduler when its
claim wins. A closed gate watches a network-paused session's turn as its
probe, so a resume that gets through opens the gate for the rest.

Prompt-failure check. The driver reports it as a flag:
`SessionSettled::prompt_failed`, read from the transport monitor's
existing `answered_error`, carried on the runner's `RunSettlement` to
`decide_failed_run`. `failed_on_the_prompt` and its parsing of the SDK's
Debug text are gone, and `PROMPT_FAILED_PREFIX` is private to the driver
again. Nothing stored changes. A turn reattached to after its connection
went has no flag and is not classified.

Chat alert. A row whose bookkeeping reason is `network_wait` shows no
retry time: its `nextRetryAt` is the time it paused.

Row subtitle. A timeline row carries no retry bookkeeping, so it cannot
tell a pause that will resume from one that gave up or is unsafe. It now
reads "Paused: network unavailable." with no promise. Keeping the
promise on healthy rows needs a new timeline payload field, left for
review.

Tests, each failing on b385cb1: the real row sequence of a failed
Claude turn leaves the gate shut and the failure uncounted, also for a
tick between those rows and the pause write; a successful result or a
prompt response opens the gate and text does not; a counted failure
after a 30 hour outage schedules a retry, and the window still ends 24
hours after the first counted failure; twenty failed probes with a
network give up, and with none never do; a failed start is counted,
hands the probe over and gives up at the cap; an empty fingerprint
keeps the doubling schedule and suppresses the wake and change
triggers; a hung probe times out; a manual resume opens the gate, or
closes it uncounted when it fails; the driver marks an answered prompt
error; a run not flagged as a failed prompt is not paused; the alert
for a `network_wait` row with a past time; and the subtitle without a
promise. The two `cloned_ref_to_slice_refs` warnings are fixed.

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Resolves the two warnings of the review of dddf059.

Timed-out probe. A probe that failed on the network only after
`PROBE_TIMEOUT_MS` escaped the bound of 20 failed probes: by then the
gate was closed, so the failure was answered `Waiting` without touching
the session's count, and its row, due at once, was probed again on the
next tick. A lone session could repeat that without limit.

The gate now remembers each timed-out probe in `timed_out_probes`, with
whether the process was suspended during the turn, before or after the
timeout. When that session later fails on the network:

- the failure is charged to its run of failed probes if the machine
  stayed awake and the last fingerprint held an address, the conditions
  of a probe that fails in time, and the twentieth gives up with
  `probe_limit_exceeded`;
- a closed gate waits at least the current probe delay from the failure
  before its next probe. The delay is not doubled again, because the
  timeout already doubled it for this probe;
- a probe another session is running is left alone.

The record is dropped when the session fails or probes again, and all
records are cleared when the gate opens, with the failure counts. A
late failure with the gate open is counted as before.

The comment on `PROBE_TIMEOUT_MS` claimed it was longer than a provider
request takes to time out. That was never measured. It now says so, and
that a probe may fail after the timeout and is then charged as above.

24 hour window. For a network pause the window is now measured only
across counted failures. `first_limited_at` was set again only when
`attempt` became 1, so the session that closed the gate (counted once),
then waited behind it for more than a day, gave up with
`retry_window_elapsed` on its second counted failure. A counted failure
now restarts the window when the pause before it was not counted, read
from the bookkeeping `reason` that pause left (`network_wait` or
`probe_limit_exceeded`). No field is added. The attempt cap is not
restarted, and counted failures with no wait between them still give up
once they span more than 24 hours. API-limit pauses are unchanged.

Tests, each failing on dddf059 except the two noted: twenty late probe
failures exhaust a session; a late failure holds the next probe for the
probe delay without doubling it, at the gate and through the scheduler
for a lone session; a late failure is charged only awake with a network
present, and is charged while another session probes without
disturbing it; a wait between two counted failures restarts the window.
Passing before and after: an opened gate forgets timed-out probes, a
late failure never brings a later probe time forward, and waits do not
restart the attempt cap. `the_network_window_runs_from_the_first_counted_failure`
asserted the old rule and is replaced by
`the_network_window_runs_across_counted_failures`.

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-20T04:55:32.063050Z a36de94 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@matt2e
matt2e merged commit 9fdfea0 into main Nov 20, 2026
10 checks passed
@matt2e
matt2e deleted the retry-on-network-or-dns-error branch November 20, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant