Skip to content

providers: retry a transient model-call failure instead of discarding the run - #62

Merged
benjaminjones merged 2 commits into
mainfrom
fix/audit-retry-transient-bridge-errors
Oct 3, 2026
Merged

benjaminjones merged 2 commits into
mainfrom
fix/audit-retry-transient-bridge-errors

Conversation

@benjaminjones

Copy link
Copy Markdown
Owner

A 12-pair audit lost all 12 calls to a single Request timed out — twice in a row. The failure propagated out of the per-pair list comprehension in audit_repo and nothing was written, so every completed pair was thrown away because one was unlucky.

The distinction the code was missing

examples retried?
permanent missing CLI, unknown model, call ceiling reached no — fails identically every time; retrying wastes time and buries the cause
transient timeout, 429, 502/503/504, connection reset yes — 3 attempts, linear backoff

The decision is made on the message, not the exception type, because the bridge raises RuntimeError for every failure mode and the type carries no information.

Two details that matter more than the retry

Every retry is counted — in USAGE and in the usage line the user actually reads. A run that silently retried is indistinguishable from a clean one, and a flaky bridge would then look like a healthy one. retries is declared in USAGE rather than materialised on first use, so no consumer has to guard against a key that only appears when something went wrong.

The call ceiling counts logical calls, not attempts. _guard() runs once before the first attempt, so --max-calls limits the questions asked rather than the packets sent — otherwise a flaky connection would silently halve a stated ceiling. There's a test for that specifically.

Tests

Six contract tests, covering both directions:

  • a timeout is retried and counted
  • a missing CLI is not retried, and its error reaches the caller unchanged
  • attempts are bounded, with the real error surfacing rather than a generic one
  • the call ceiling is never treated as transient — which would turn a deliberate limit into a delay

Then exercised against the actual failure shape: a 12-call run with a timeout on the fifth call now completes, reporting 12 calls · 1 retry.

ruff check . passes · pytest 342 passed (336 + 6)

🤖 Generated with Claude Code

benjaminjones and others added 2 commits September 30, 2026 20:21
… the run

A 12-pair audit lost all 12 calls to a single `Request timed out`, twice in a
row. The failure propagated out of the per-pair list comprehension in
audit_repo and nothing was written, so every completed pair was thrown away
because one was unlucky.

The fix distinguishes two failure modes that were previously treated alike. A
missing CLI, an unknown model or a reached call ceiling fails identically on
every attempt -- retrying wastes time and buries the cause. A timeout, a 429 or
a 5xx is a property of the moment. Only the second kind is retried, three
attempts with a linear backoff, and the decision is made on the message rather
than the exception type because the bridge raises RuntimeError for everything
and the type carries no information.

Two details that matter more than the retry itself.

EVERY RETRY IS COUNTED, in USAGE and in the usage line the user actually reads.
A run that silently retried is indistinguishable from a clean one, and a flaky
bridge would then look like a healthy one. `retries` is declared in USAGE rather
than materialised on first use, so no consumer has to guard against a key that
appears only when something went wrong.

THE CALL CEILING COUNTS LOGICAL CALLS, NOT ATTEMPTS. `_guard()` runs once before
the first attempt, so --max-calls limits the questions asked rather than the
packets sent; otherwise a flaky connection would silently halve a stated
ceiling. A test pins that.

Six contract tests cover both directions: a timeout is retried and counted, a
missing CLI is not retried and its error reaches the caller unchanged, attempts
are bounded with the real error surfacing rather than a generic one, and the
ceiling is never treated as transient -- which would turn a deliberate limit
into a delay. The whole thing was then exercised against the actual failure
shape: a 12-call run with a timeout on the fifth now completes, reporting 12
calls and 1 retry.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@benjaminjones
benjaminjones merged commit d051baf into main Oct 3, 2026
4 checks passed
@benjaminjones
benjaminjones deleted the fix/audit-retry-transient-bridge-errors branch October 3, 2026 19:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant