Skip to content

perf(staged): add the worktree while a transfer from Blox is still capturing - #1089

Merged
matt2e merged 4 commits into
mainfrom
transfer-from-blox-adding-the-worktree-could-start-earlier
Nov 18, 2026
Merged

matt2e merged 4 commits into
mainfrom
transfer-from-blox-adding-the-worktree-could-start-earlier

Conversation

@matt2e

@matt2e matt2e commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Summary

When moving a project from a Blox workspace to this machine, each branch's local worktree was added only after the branch had been snapshotted, downloaded and landed, and the "Adding the worktree" step also covered a cold capture of the new worktree's shell environment. This PR adds the worktree early, while the transfer is still capturing, and then hardens that path against the problems review turned up.

  • Early worktree add. Each leg picks a start commit the clone already has (the inspected head, origin's copy of the branch, its upstream, or the base), records the worktree in the undo list, and adds it detached with hooks off on a scoped thread that also starts the shell-environment capture. Once the branch lands, git checkout <branch> moves the worktree onto it. Any failure, a missing start commit, or a wrong Git LFS guess falls back to the previous late add at the same path.
  • Safe removal. The shell-environment cache gains wait_out_and_invalidate, and the shared worktree removal routine waits out any in-flight capture before deleting, so rollback cannot pull a directory out from under a starting login shell. Removal also handles the initializing lock left by a killed git worktree add (force twice, then unlock and prune).
  • Hooks fire once, as for a new worktree. Hooks are off for both the add and the switch, and post-checkout is fired once afterwards via git hook run with the null id, the head and 1. A git older than 2.36 plans no early worktree.
  • Capture after the hook. When a post-checkout hook would run on the branch, the early capture is waited out and dropped before the hook fires, so the finish captures after it. The stock Git LFS hook is exempt so LFS repos keep the head start.
  • Config-named hooks. Hooks named via hook.<name>.event (git 2.54+) run regardless of core.hooksPath=/dev/null, so a clone with such a post-checkout plans no early worktree. Hooks in the default or an absolute hook directory are detected from the clone before the add so no doomed capture is started; a relative core.hooksPath is still decided once the worktree is on its branch.

Progress phases are unchanged. Each leg logs its early add, switch, environment wait and finish durations, whether it fell back, and whether the early capture was skipped or dropped.

Tests

New tests in orchestrator_tests/early_worktree.rs, git/post_checkout_hook/tests.rs and shell_env/wait_out_tests.rs cover the overlap of the add with the snapshot (gated both ways), fallbacks, the start-commit choice, the invalidation rule, hook arguments compared against a real git worktree add, a hook killed mid-add leaving a locked entry, the LFS hook texts, detection of default/absolute/relative/config-named hooks from the worktree and the clone, and the cache wait against a real held shell. A test-only OwnConfigOnly guard keeps the machine's git config out of these tests.

🤖 Generated with Claude Code

matt2e and others added 4 commits November 17, 2026 14:43
…pturing

Moving a project from a Blox workspace to this machine, each branch's
local worktree was added only after the branch had been snapshotted,
downloaded and landed, and the step shown as "Adding the worktree" also
covered a cold capture of the new worktree's shell environment.

The worktree needs only a commit to check out, and the clone already
holds one nearby. Each leg now picks it before its capture starts: the
head the inspection saw when the clone has it, else origin's copy of the
branch, its upstream, or the base. It records the worktree in the undo
list, then adds it detached at that commit, with hooks off, on a scoped
thread that also starts the shell-environment capture. The snapshot, the
download and the landing run on the leg's thread as before. Once the
branch has landed, `git checkout <branch>` moves the worktree onto it,
which rewrites only the files that differ and fires `post-checkout`
once, and the finish runs as it did.

The early worktree is an optimisation with a complete fallback. With no
commit to start from, an early add or switch that fails, or a guess
about Git LFS (from the inspection's counts) that the snapshot
contradicts, the leg clears the early worktree and adds one on the
branch at the same path, as before. A capture that fails or is cancelled
returns its own error; the thread is always joined first and rollback
removes the recorded worktree, so no cleanup path is new. Only transfers
to local take this path.

The prewarmed environment was captured at the start commit, so it is
dropped after the switch when the two commits differ under `bin/` or in
a root-level marker file the cache keys on, and the finish recaptures.
The path's cached environment is also dropped before the prewarm, since
anything cached there belongs to an earlier worktree.

Progress phases are unchanged. Each leg now logs its early add, switch,
environment wait and finish durations and whether it fell back.

Tests cover the worktree being added while the snapshot runs, a snapshot
failing after the early add, a head that moves after the inspection, the
early add failing, the LFS guess being wrong, the undo list naming the
worktree at the first snapshotting event, the start-commit choice, the
invalidation rule, and the detached add and switch firing one hook.

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…o hooks

A transfer from Blox adds each branch's worktree while the capture still
runs and starts a shell-environment capture in it. Review of that change
found four things to fix.

Rollback could delete the worktree while that capture's login shell was
still starting inside it, since the leg joined the add but not the
capture. Whatever the shell wrote could stop the removal or bring the
directory back. The shell-environment cache has no way to stop a capture,
so it gains `wait_out_and_invalidate`: wait for the capture in flight for
a directory, never starting one, then drop what is cached. The shared
removal routine calls it first, which covers rollback, startup recovery
and the fallback alike, and replaces the fallback's own wait. A cancel
during the prewarm now takes up to one capture longer. A capture made
stale by the switch is waited out the same way instead of being dropped
while it runs, so no second shell starts beside it and none is left where
a later removal cannot wait for it.

A `git worktree add` killed part way leaves an entry locked with the
reason `initializing`. A singly forced removal refuses it, a prune keeps
it, and every later add at that path fails. The removal routine now
forces twice, and unlocks before pruning when that fails too.

`post-checkout` fired from the switch, so it was told the start commit
was the previous head, and for a fully pushed branch got the same commit
twice. A hook that sets up a new worktree on seeing the null id did
nothing. Hooks are now off for the switch as well, and the hook is fired
once afterwards by `git hook run` with what `git worktree add <path>
<branch>` passes: the null id, the head and `1`. A hook that fails there
fails the leg, as it fails `git worktree add`, rather than falling back
to an add that would fire it again. `git hook run` needs git 2.36, so a
leg whose git lacks it plans no early worktree and adds late as before.
One difference remains: git exports `GIT_DIR` to a hook it fires inside
a linked worktree, where `git worktree add` clears it.

The path with no early worktree now also drops what is cached for its
path before adding, as the early path does.

Tests: a gate holds the early add until the snapshot has started and the
snapshot until the add has finished, so the overlap is proven rather
than only the missing dependency. Hook tests record the arguments on
both paths and compare them with a worktree git added itself, cover the
equal-commits case, a failing hook and a git without `git hook run`. A
real `git worktree add` is killed in a smudge filter to reproduce the
locked entry, with and without its directory, and removal plus a re-add
at the same path are checked. The shell-environment calls, no-ops under
test, are now recorded per watched directory with whether the path
existed, which pins the wait before the removal on rollback and on the
fallback, and the wait before a recapture. The cache method is tested
against a real held shell.

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…checkout

A transfer from Blox adds each branch's worktree early and starts a
shell-environment capture in it, then fires `post-checkout` once the
worktree is on its branch. Review of that found the capture ran before
the hook. A hook that sets a new worktree up, by allowing its `.envrc`,
trusting a tool-versions file or writing either, did so after the
capture's shell had started, and the finish then used that capture,
which stays cached for up to an hour. A worktree added late is first
captured after `git worktree add` has fired the hook. The hook also ran
beside the capture's login shell in the same directory.

Once the worktree is on its branch, the leg now asks which
`post-checkout` hooks git would run there. When one would, it waits out
the early capture and drops it before firing the hook, so no shell of
that capture is in the directory while the hook works and the finish
captures after it: the order the late path has. A repo with no such hook
keeps the capture. The check that the two commits differ in what a
capture reads shares that one wait, now before the hook instead of after
it.

`git::post_checkout_hooks_for_worktree` answers the question the way
`git hook run` will: the hook directory's file from `git rev-parse
--git-path`, which follows `core.hooksPath` from the worktree, counted
when it is there and executable, and git's own `git hook list` where it
has one, which is where git also runs hooks named in config. What it
cannot tell counts as a hook.

One hook is exempt: the `post-checkout` that `git lfs install` writes,
so Git LFS repos keep the head start. `git lfs post-checkout` only sets
the write flag of lockable files. The hook is recognised by its three
lines: `#!/bin/sh`, the guard that reports a missing `git-lfs` and exits
2, and `git lfs post-checkout "$@"`. Git LFS has reworded the report
several times, so its text is free as long as it stays one double-quoted
string with nothing the shell would run. Anything added or changed, or a
second hook from config beside it, makes it any other hook.

A hook manager that owns the machine's `core.hooksPath` and installs a
`post-checkout` for every repo counts as a hook, since what it goes on
to run cannot be known. Repos under one lose the head start on the
environment unless they set a hook directory of their own.

The doc comment on `run_post_checkout_as_for_a_new_worktree` named
`GIT_DIR` as the one difference from a late add. It now states the
second: a relative `core.hooksPath` is resolved from the new worktree,
so the branch's copy of the hook runs where `git -C <clone> worktree
add` runs the main checkout's, and a branch without that directory
fires no hook, silently. The behaviour is unchanged.

Each leg's log line says when the early capture was dropped, how long
the wait took and why.

Tests: with a hook, the capture is waited out and dropped while the hook
has not yet run, and the finish's wait comes after it; firing the hook
first fails that test. With an empty hook directory, with a hook git
would ignore for not being executable, and with the stock Git LFS hook,
the capture is kept. The detection is tested against a real linked
worktree for each case, a relative `core.hooksPath` whose directory only
one branch has, a hook named in config, the four texts Git LFS has
shipped, edited copies of the stock hook, and what `git lfs install`
writes on the machine running the tests. The shell-environment recorder
can note whether a file existed at each call, which is how the hook is
placed among them. Tests that look at those calls give the clone its own
hook directory, so a machine-wide one does not decide them.

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…-checkout hook

A transfer from Blox adds each branch's worktree early with the hook
directory turned off, switches it onto the branch the same way, then
fires `post-checkout` once with `git hook run`. Review of that found one
defect and two things to tighten.

`core.hooksPath=/dev/null` only turns the hook directory off. Git 2.54
and later also run hooks that config names (`hook.<name>.event` with
`hook.<name>.command`), wherever the hook directory points. Such a
`post-checkout` fired at the detached add, told of the start commit,
again at the switch, told the worktree came from there, and a third time
on request. A leg now asks the clone's effective config, with `git
config --get-regexp` over the `hook.<name>.event` keys, before it plans
an early worktree. When any of them is `post-checkout` it plans none and
adds the worktree once the branch has landed, as before this branch, so
git fires the hook once with the null id. An event counts even if a
later empty value or `hook.<name>.enabled` takes it back, and on a git
too old to run it.

The check that decides whether the early capture of the shell
environment must be dropped read `git hook list` and took any failure
for "no hooks named in config": no hooks, a git without the subcommand,
a config git could not parse, and a child that never ran all looked the
same. It now uses the same config query, which every git answers. Only
exit 1 with nothing printed is "none". Any other exit, a killed child,
or a git that cannot be spawned counts as a hook, so the capture is
dropped. `cli::run_output` returns the exit status with the captured
environment, as `run_lite_output` does with the inherited one.

A hook that may do anything was only looked for once the worktree was
on its branch, so the early add always started a capture that such a
hook then cost: a login shell, one of the four capture slots, and a
wait before the hook. `git::post_checkout_hooks_to_come` now answers
from the clone before the add. The default hook directory and an
absolute `core.hooksPath` are the same from every worktree, so a hook
there is already known, and the early add then starts no capture. The
worktree is still added early. A relative `core.hooksPath` makes the
hook the branch's own file, so that case alone is still decided once
the worktree is on its branch, with the wait-out as before. The check
on the branch stays in every case, and still waits out whatever capture
is in flight before a hook fires. Each leg's log line says when no
early capture was started.

The clone's config is read as its main checkout reads it. A conditional
include that only the new worktree matches is not seen before the add.

Tests no longer depend on the machine's git config. A test-only guard,
`OwnConfigOnly`, keyed by directory like the shell-environment recorder,
makes the git that `git::cli` runs for repos under it read no global or
system config. The early-worktree tests and the hook-detection tests
hold one, so a hook manager's `core.hooksPath` or a hook named in the
user's config no longer decides them, and the default hook directory is
exercised. The per-test empty hook directory is gone.

Tests: a clone with a config-named hook that logs every invocation gets
its worktree added on the branch, starts nothing early, and logs one
line with the null id; with the plan check removed that test fails. A
hook in an absolute hook directory keeps the early add, starts no
capture, and runs before the finish's. A hook committed on the branch
under a relative `core.hooksPath` keeps the earlier order: prewarm,
wait-out, hook, capture. The detection is tested from the worktree and
from the clone for the default directory, an absolute one, a relative
one and a config-named hook, and the two must agree where the directory
is shared. The exit-status reading is tested over "none", other events,
a taken-back event, exits 128, 129 and 3, an exit 1 with a complaint or
with output, and a killed child. A repo whose config git cannot parse,
and a path that is no repo, count as having a hook.

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-18T03:55:24.954504Z 5c6e7df PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5c6e7df3a9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +239 to +244
let mut args = vec!["diff", "--name-only", "--no-renames", start, head, "--"];
for input in crate::shell_env::env_root_inputs() {
args.push(input);
}
match crate::git::cli_run(clone, &args) {
Ok(changed) => !changed.trim().is_empty(),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Invalidate early captures when indirect environment inputs change

When the remote branch changes a file sourced by an unchanged environment marker, this whitelist incorrectly retains the environment captured at the starting commit. For example, an unchanged .envrc containing . ./project-env.sh can export different tool paths or variables after project-env.sh changes, yet this diff is empty because that file is outside env_root_inputs(). The finish script and subsequent commands then use the old environment, potentially for the cache's one-hour TTL, whereas the previous late capture read the landed commit. Conservatively invalidate when commits differ unless all dependencies of the environment configuration are known.

Useful? React with 👍 / 👎.

@matt2e
matt2e merged commit e67b888 into main Nov 18, 2026
8 checks passed
@matt2e
matt2e deleted the transfer-from-blox-adding-the-worktree-could-start-earlier branch November 18, 2026 04:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant