feat: compare backends side by side — one prompt, 2–4 forks, keep the best

[saucam]

Closes #357. Builds on #363 (fork from a turn), #362 and #361, all merged.

What

Send one prompt to 2–4 backends/models at once and compare what each did, then keep the best. Each branch runs in its own fork and git worktree, starting from the same conversation and files: the latest point, or after an earlier turn.

For each branch you see:

• status, including needs your approval (branches ask in their own session);
• its reply to the prompt;
• files changed (+/−, paths);
• cost and time.

Keep one, optionally destroying the rest.

• Web: "⚖ compare" in session controls, "⚖ compare from here" on a message. The dialog shows the branches in columns, with open, keep, and keep + discard others.
• CLI:
  • codeoid compare run <s> --with claude,codex:gpt-5.5 [--at N] [--no-wait] <prompt…>
  • codeoid compare show <id> [--wait]
  • codeoid compare ls <s>
  • codeoid compare keep <id> <branch> [--discard-others]
• TUI: /compare <targets> [--at N] <prompt>, /compare, /compare keep N [--discard-others].
• Protocol: session.compare, compare.get, compare.list, compare.keep → compare.state / compare.list.result.

How (backend-agnostic)

It is pure orchestration over session.fork and send, so it works with every current and future backend. The comparison record lives in the store (compare_runs).

When a branch's compared turn settles, its result is frozen from codeoid's own records:

• the reply comes from the canonical history for that turn only;
• the files come from the start→end snapshot numstat;
• the cost and time are that turn's totals.

Carrying on in a branch later never changes the comparison, and reads stay cheap.

Safety and robustness

• Refused up front, leaving nothing behind:

  • a model the backend doesn't have, or one that isn't a model id;
  • a parent that's still working;
  • a folder that isn't a git repo, since every branch needs its own worktree.

  Any failure while forking rolls back the branches made, along with their git branches.

• Request handling: the request answers as soon as the branches exist, and the sends run without the client waiting.

• Keeping: one keep per comparison, and only a finished branch can be kept. A slow settle can't undo a keep.

• Cleanup: comparisons are deleted with their session.

• Forks: they now carry the parent's pack, re-resolved, so a role's tool deny survives. A pack that can't be re-activated refuses the fork.

• Unrelated fix in codeoid approve: it always sent an empty approval id, which the daemon rejected. It now shows the pending tool call in full and asks before approving (--pick N, --yes).

Testing

• src/tests/compare.test.ts (12) uses the real SessionManager with mock backends. It covers:
  • replies, files, and cost per branch;
  • comparing from an earlier turn;
  • the same backend with two models;
  • a failing branch, with no inherited reply;
  • frozen results;
  • keep and discard, and the second-keep refusal;
  • the up-front refusals with no leftovers (model, flag-like model, busy parent, non-git);
  • destroy and restart cases;
  • the keep-vs-slow-settle race;
  • no resurrection after the parent is destroyed;
  • gone and never-started branches.
• src/tests/terminal-compare.test.ts (7) and web CompareModal.test.tsx (4) cover the clients.
• Full gate passes: typecheck, biome, bun test (2931 pass), web vitest (593 pass), build.
• Live E2E, on a real daemon with real backends, through the CLI:
  • claude vs codex:
    • Approvals are surfaced, and codeoid approve shows the exact command and asks first.
    • claude's result: its own reply, fib.py +8, $0.18, 14s.
    • codex reported its own failure honestly. Its sandbox can't start inside this test environment, which is unrelated to compare.
    • Keep with discard removed the other branch and its worktree, and a second keep was refused.
  • claude vs claude:sonnet from turn 1: both branches started from that turn's files and each appended its line, while the parent's files stayed untouched.

Audits

Two deep rounds, each covering correctness and security; all fixes are folded into the two fix: commits.

One finding is noted but not changed: a follow-up sent to a branch during its compared turn joins that turn, as any mid-turn message does.

🤖 Generated with Claude Code