Repository navigation
fix: reuse writable uppers and reconcile upstream through 04f8786 - #2
Closed
William Zhang (17zhangw) wants to merge 132 commits into
Closed
William Zhang (17zhangw) wants to merge 132 commits into
William Zhang (17zhangw) wants to merge 132 commits into
Conversation
Propagate the optional override through the snapshot client and OverlayBD runtime while preserving endpoint-based auto-detection by default.
Co-authored-by: husp <husp@inesa.com>
Co-authored-by: nzhu <nick.zhu@booking.com>
Use synchronous positional file I/O for ordinary VirtualFile calls while preserving the caller-provided io_uring fast path. Remove ImageService-owned ring pools and transient fallback workers. BREAKING CHANGE: LocalFile constructors no longer accept an IoRingHandle, and ImageService::io_ring was removed. Co-Authored-By: GPT-5 <noreply@openai.com>
Make io_uring support optional in storage-util and OverlayBD so portable builds can use the ordinary VirtualFile path without pulling in Linux io_uring code. Keep ublk consumers on the existing context-aware path by enabling the feature explicitly. Co-Authored-By: GPT-5.6-sol <noreply@openai.com>
Moving LocalFile to std::fs::File dropped the implicit spawn_blocking that tokio::fs::File gave set_len and sync_all, so sync and truncate began running inline on the caller thread. On a ublk queue thread (current_thread runtime driving a LocalSet) an inline fsync stalls every in-flight I/O on that queue. Route them, plus discard and evict_range/evict_all, through a new LocalFile::offload helper that hands the closure an owned Arc clone to satisfy spawn_blocking's 'static bound. The helper also absorbs the duplicated join-error boilerplate in read_at and write_bytes_at. discard matters most: FALLOC_FL_PUNCH_HOLE is real filesystem work, not a hint, and a guest fstrim reaches it in a loop via discard_range. It was inline before the io_uring decoupling too, so this is not a regression, but it is the same failure mode on a guest-triggerable path. Its added yield point only widens an existing window, since discard_range takes the index write lock after the call returns. size stays inline on purpose: a bare fstat off the cached inode is cheaper than the spawn_blocking round trip that would offload it. The split now follows one rule -- offload syscalls whose latency has no upper bound, keep the cheap bounded metadata ones (fstat, lseek, xattr) inline. write_at and read_at_into still block; their borrowed buffers cannot satisfy spawn_blocking's 'static bound without an ownership-passing trait change, which the existing TODO tracks. Also close a gap in the io-uring feature this branch introduced. The feature and the target are independent axes in Cargo, so `--features io-uring` was expressible on non-Linux even though the io-uring crate is only a dependency for Linux targets -- it produced a pile of unresolved-crate errors inside io_ring rather than saying what was wrong. Reject it with compile_error! in storage-util. Guarding it there is what keeps `cfg(feature = "io-uring")` sufficient on its own at every use site: the feature now implies a Linux target, so no call site has to repeat target_os = "linux". Co-Authored-By: Claude <noreply@anthropic.com>
…pled Capture/upload side of startup-memory acceleration (manifest variant). At snapshot publish (OSS backend, feature enabled), a throwaway recording VM re-resumes the captured snapshot on a dedicated memory device while a daemon-side recorder observes first-touch pages in a bounded window. The recording is spawned alongside the publish layer uploads instead of ahead of them, and a detached continuation expands the trace into a compact range table sorted by each range's earliest first-touch rank, uploads just that list as memory-startup.pack — zero layer bytes read on the publish path — and attaches the sha256-bound descriptor to the committed record afterwards. Publish itself never waits on any of it (measured ~16s -> ~6s). Graceful shutdown stops new continuations and drains in-flight ones (bounded), canceling only what outlives the drain; daemon abort RPCs are time-boxed so a dying daemon cannot hang the exit. POSIX backends, disabled configs, and any recording/build failure publish normally without a descriptor.
Consume the v3 manifest at resume: probe the magic, gate on the committed sha256, translate pages to final-object blocks with the shared planner (tar unwrap, zfile jump-table extents, merged LSMT index), and refill them through the same batched source-read path background downloads use. Adjacent blocks are merged into capped runs (large requests); runs are delivered sorted by each run's earliest first-touch rank so boot-critical blocks land first without giving up request efficiency.
…ai#250) Follow-up to the dynamic sandbox-ID completion: `aenv start`, `aenv template watch`, and `aenv template delete` now complete live templates and snapshots from the existing list APIs. Names and aliases are offered ahead of IDs, since names are what users type and IDs are only a fallback for resources that have none. `aenv start --cold` takes an external OCI image reference, so it offers no local resources. A candidate provider gets no context about the rest of the command line, but the completion callback runs as `COMPLETE=<shell> aenv -- <words...>`, so the typed words are in our own argv; scanning it lets the provider return before credentials or the network are touched. Candidates are sorted within their groups for a stable order and de-duplicated, since a template can list the same string as both a name and an alias. Each resource kind contributes at most 100 candidates, so many templates cannot crowd snapshots out of `aenv start`. Co-authored-by: nzhu <nick.zhu@booking.com> Co-authored-by: Claude Code <noreply@anthropic.com>
…runtime Each ublk device queue runs its own current_thread tokio runtime that is dropped on device teardown, while opendal (GLOBAL_REQWEST_CLIENT) and the cached operators share one process-wide reqwest connection pool. hyper spawns per-connection driver tasks onto whatever runtime polls a request, so pooled connections ended up owned by a random device's runtime: reads from other devices coupled to that device's scheduling, and device teardown killed connections other devices were actively using. Introduce RuntimeDispatchFile and route all remote network I/O through a fixed runtime owned by ImageService: - open_backend_source_with_size runs the whole open (including the eager size fetch in RegistryFsV2::open) on the pinned runtime and wraps the opened OSS/registryfs files, so no request or steady-state read is ever polled on a caller's runtime regardless of what the open path does; - the lazy remote-runtime construction and background-download submission run there too, so the cache's internal tokio::spawn calls (download scheduler, eviction/checkpoint workers) bind to it without the cache internals needing to know; - drop the custom streaming read_at_into override on OssFile: the VirtualFile trait default (read_at + copy) now serves those callers.
Support `POST /v2/sandboxes` and `POST /v2/sandboxes/{sandboxID}/connect`
Add setup, start, resume, and finish commands with a bundled executor template. Run native Codex against an AgentENV sandbox using its existing configuration and authentication. Include project transfer, workspace RPC routing, lease renewal, result recovery, and regression tests.
Document setup, sandbox sessions, resume, and result recovery. Add links from the use-case index and aenv CLI guide.
…TTPS With an https:// API URL, aenv build panicked when it opened the BuildKit tunnel: "Could not automatically determine the process-level CryptoProvider from Rustls crate features". reqwest's rustls feature enables aws-lc-rs and ureq's tls feature enables ring, so rustls cannot choose a default, and tokio-tungstenite's TLS connector relies on the default. main now installs aws-lc-rs, the provider reqwest uses, before anything opens a connection. The new test opens a wss:// connection to a listener that hangs up; without the provider it panics, with it it fails like any bad connection. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… rootfs directory Vendor a minimal containerd proxy snapshotter that bind-mounts an existing host rootfs directory as a container's root filesystem — no image pull, no layer unpacking, zero copy. Ships a 'register' command that maps a placeholder image's chain ID to the directory, with optional -image-metadata to preserve the original image's ENV/ENTRYPOINT/CMD. Co-authored-by: guozy18 <guozy22@mails.tsinghua.edu.cn>
Avoid materializing lower-layer mappings that close_seal immediately discards. Query the mutable upper directly while preserving virtual-size clipping. Signed-off-by: Park.Jiyeon <jiyeonnn2@icloud.com>
Reserve the all-ones physical offset for unbacked zero segments. Normalize legacy lower zeros at the readonly index boundary while preserving backed-zero offsets during writable index replay. Update clipping, merging and compact output to honor the marker. Hybrid discard/write reuse is left for the next step. Co-Authored-By: GPT-5 <noreply@openai.com> (cherry picked from commit f6075eb)
Preserve physical offsets when discarding allocated upper ranges. Mark only gaps as unbacked zeros. Rewrite backed zeros in place and append non-zero mappings so the transition survives reopening. Keep existing write, chunking and error-handling behavior. Cover partial ranges, lower masking, repeated rewrites and io-uring I/O. Co-Authored-By: GPT-5 <noreply@openai.com> (cherry picked from commit 5694dbe)
Retain durable create claims, fail-closed preservation and stop proof, non-destructive snapshot copies, reference-aware checkpoint retention, tools/runtime identities, and renewable GCS credentials while integrating upstream volume, network, API, scheduler and storage changes. Transfer prepared volumes to their sandbox owner before backend startup, keep immutable per-capture volume configs, and retain deletion claims until metadata removal completes. Builder validation: 966 AgentENV library tests and 385 OverlayBD library tests passed; explicit sustained reuse, ENOSPC and cache concurrency tests passed. Workspace lint, optional I/O paths, VM migration and release acceptance remain follow-up gates.
Select a new hybrid rootfs upper only for immutable log-mode memory checkpoints with consistent rootfs and ublk descriptors. Keep the persisted checkpoint, RAM/VM state, tools and runtime identities unchanged; reject existing writable source uppers. The policy defaults off and does not affect fresh/cold boots or read-only, sparse, hybrid and startup-pack recording launches. Three builder policy tests pass, including unchanged checkpoint bytes and serialized metadata, opt-in behavior and refusal paths. Actual VM/process continuity remains a required acceptance gate.
Keep snapshot and fork source handles, metadata and resource ownership when terminal cleanup cannot confirm a stop. Leave deletion in its stop phase so a normal retry must obtain stop proof before releasing volumes or artifacts. Cover both terminal failure paths with stop-failure and retry tests.
12 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Superseded by #3. The authoritative branch is
codex/upstream-patch-series, directly based on upstream04f87864da78d3fb3c2f96bf6a2b719017edc843. This PR, branch, commits and successful signed build are preserved as references; its candidate will not be staged.What
Fix cold-boot upper-mode selection and integrate the complete upstream history through
04f87864da78d3fb3c2f96bf6a2b719017edc843, retaining OneWill preservation behavior. Add explicit memory-restore migration from a pure-log checkpoint to a fresh hybrid upper.Why
The fork forced cold boots to use append-only log uppers. Repeated updates could exhaust the worker state disk. Hybrid reuse must include backed-zero and discard/rewrite correctness; changing the default alone cannot migrate existing computers.
Related issue
User-directed storage incident remediation and upstream reconciliation; no separate issue.
Scope and non-goals
Includes the full upstream target, rather than only its three storage patches. No dev or production deployment, existing-computer tools upgrade, live-file conversion, periodic snapshot workaround, or historical recovery deletion.
Design and behavior changes
40ff218fixes cold-boot policy; merge91a97e6integrates the exact upstream target. Subsequent commits add lint corrections, explicit migration, terminal capture stop retry, and failed-launch/missing-handle retention.ublk.overlaybd.restore_log_upper_as_hybrid. It changes only the new rootfs/backend upper modes before allocation, refuses existing writable files and inconsistent metadata, and leaves the immutable checkpoint untouched. Default is false; OneWill policy is a separate Hambody change.Compatibility and operations
Validation
make fmtequivalent:cargo fmt --all -- --checkmake clippyequivalent:cargo clippy --locked --workspace --all-targets --all-features -- -D warningsmake test-unitcomponents (nonroot builds; explicit capability runner for privileged binaries)go test ./...; Go formatting and vetcargo adev codegen serverLinux validation on onewill-builder, nonroot Rust builds/tests:
MinIO: all 7 tests passed using a local source-built mirror because Quay returns HTTP 401. Source is the exact pinned release tag, commit
07c3a429bfed433e49018cb0f78a52145d4bedeb; image IDsha256:04248e2077fa460405edf5ab834442406d4fdad8eb73dad6d18bedd0e13b92d3.AENV_TEST_MINIO_IMAGEchanges only the test image repository while retaining the release tag. CLI: all 110 tests passed serially and on subsequent normal parallel runs; one earlier parallel run had an intermittent lock-release assertion failure. Capability runner, service-install, paused-recovery boundary, and CLI packaging/install checks passed. Linux-cap: 3 passed; envd library has no tests. All compile/test operations were on the builder; root-only installation fixtures use sudo internally.Remaining gates: Real RAM/process continuity, checksummed files, dirty Git, SQLite, trusted SSH, published ports, tools retention, VM fault injection, checkpoint/rollback allocation, signed release and staging-2 remain unverified.
Risks and reviewer notes
DRAFT: this is a review milestone, not release approval. The audited failed-launch and missing-handle capture cleanup paths now retain resources until stop is proven, with graceful deletion retry tests. Concurrent delete releases its mutex while waiting for capture so terminal stop retry cannot deadlock. Total disk usage is not proven bounded: mapping-index growth is separate from data extent reuse. Actual guest migration has not yet been demonstrated. Hambody must supply the shared API key required by upstream authentication and opt into migration in its separately reviewed release change.
Checklist