diff --git a/CLAUDE.md b/CLAUDE.md index 6267a495..cd5ae2f8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,2983 +1,245 @@ # CLAUDE.md -## Project Overview +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. -**lightcone-cli** is Lightcone Research's execution layer for ASTRA -(Agentic Schema for Transparent Research Analysis). It ships the `lc` -executable — an agent-agnostic CLI; it bundles no agent-specific skills, -hooks, or plugins. +## What this is -- **ASTRA** = pure specification: schema, validation, prior insights & - findings, evidence verification, helpers, minimal CLI -- **lightcone-cli** = execution layer: project scaffolding, execution, - environment identity, hermeticity enforcement +**lightcone-cli** ships `lc`, the execution layer for ASTRA (`astra.yaml` +analysis specs, spec 0.1). The split with `astra-tools` is enforced in +code: what a spec *means* (validity, universes, declared analyses, pins, +`when:`, the recipe placeholder grammar, where a result and its record +go, and whether a record still describes the analysis) is answered by +`astra.api` — `plan` and `check`; lc is the *runner*: it only adds +execution — scaffolding, environment identity, sandboxing, storage, and +writing each result's record. If a spec's meaning looks wrong, the fix +belongs in astra-tools, not here. -lightcone-cli depends on ASTRA. The `astra` CLI handles spec operations; -the `lc` CLI handles execution. +ASTRA 0.1 is not on PyPI yet: `pyproject.toml` pins astra-tools and +astra-spec to git commits (`[tool.uv.sources]`, `override-dependencies`). +Move both together. -That split is enforced in code, not just described: everything about what -a spec *means* — scoping, `from:` references, conditional outputs, -universe resolution, the recipe placeholder grammar — is answered by -`astra.resolve` and validated by `astra.validation` before lc acts on it. -lc's whole ASTRA surface is ten functions; see Key Invariants (layer 4). +The CLI is agent-agnostic: it bundles no skills, hooks or plugins, and a +verb is run by an agent more often than a person — so no interactive +prompts, ever. -## ⚠️ This repository is a clean rebuild in progress - -The codebase is being **re-added layer by layer** on top of the design -spec: - -> **`../lightcone-cli/docs/design/execution-environment.md`** — *"the -> locked environment is the execution environment"*, v6.1. Read it before -> adding anything — but read it as a **reference, not gospel**: the -> rebuild deliberately drifts from it as implementation teaches better -> answers, and where this file's Recorded decisions disagree with the -> spec, the decisions win. It lives in the sibling checkout for now -> (branch `redesign_prototype`), alongside its decision records -> (rationale, substrate tradeoffs, hermeticity enforcement, the v6 -> review). It stays in the sibling checkout and is **dropped when the -> rebuild completes** (decision, 2026-08): the design records are not -> imported into this repo — this file carries the decisions. - -The pre-rebuild codebase (Snakemake shim, authored Containerfiles, -`container:` in `astra.yaml`, vendored dask executor plugin, WRROC export) -was stripped deliberately. Functionality comes back **one layer at a -time**, each layer landing with its own tests and dependencies — never -speculatively. - -### Layers - -| # | Layer | State | -|---|-------|-------| -| 1 | **Project scaffolding** — `lc init` | ✅ **done** | -| 2 | **Environment layer** — `env_version`, lock scan, manifest schema | ✅ **done** | -| 3 | ~~The `lc` entrypoint — launcher~~ | ❌ **removed by decision** (2026-08) — see Recorded decisions: the host `lc` is the engine, so there is nothing to delegate to | -| 4 | **Fabric** — `lc materialize`, worker sequence, mid-run relock gate | ✅ **done** | -| 5 | **Sandbox layer** — Landlock / Seatbelt, exec-shim, denial UX, `lc run` | ✅ **done** | -| 6 | **Container hatch** — `[tool.lightcone.image]`, `lc build`, OCI runtimes as the exec boundary, the image archived in the dataset | ✅ **done** | -| 7 | **Compute** — explicit local/Slurm allocation and podman-hpc | 🔶 **landed; Perlmutter spike pending** — hub/GKE and Cloud Build deferred to their own layer | -| 8 | **Publication view** — the RO-Crate converged by materialize, foreign writes stale by history; **no `lc verify`, no `lc export`, by decision** | ✅ **done** | -| 9 | **App** — `lc view`: the analysis on one page, served by Jupyter | 🔶 **landed** | - -`lc status` landed with the invalidation model rather than at layer 8: -once an output can be *behind*, something has to say which ones are, and -the verb is the same classification walk `--check` already does. - -Layer 5 landed **out of order**, ahead of 2–4: `lc run` is the spec's -*probe* verb, and a probe has no output, so it needs neither manifests -(layer 2) nor the fabric (layer 4) — only project discovery, which came -with it. That makes it the smallest honest consumer of the exec boundary, -and the boundary is what layer 4 will then plug recipes into. - -The spec's §11 (Migration) is the reference ordering; the table above is -the working map, and the spec is a reference the rebuild deliberately -drifts from — reversals land as Recorded decisions here rather than -waiting on a spec rewrite. **Layer boundaries are also dependency -boundaries** — a dependency enters `pyproject.toml` with the layer that -needs it, not before. - -### Rules while rebuilding - -Each of these has been asked for in review at least once; none is optional. - -- **Never reference the design spec in code or comments.** No `spec §7`, - no section numbers, no "the spec says". Code and its comments must - stand on their own; design rationale lives in the design documents. - (This file is the exception — it is *about* the design.) -- **No backward-compatibility code.** Nothing exists to honor the - behavior of an older CLI, an older wire format, or trained fingers. - If old behavior isn't promised, don't guard, version, or migrate it. -- **No foreshadowing.** No code, comment, flag, or user-facing message - may mention a verb, layer, or feature that does not exist yet. The - codebase is consistent with the project *at this point in time*. -- **No escape hatches around guarantees.** A feature that enforces - something ships without a flag to turn the enforcement off. -- **Prefer literal behavior over invented convenience.** The current - directory is the project root; erroring beats walking up or guessing. -- **Nothing waits on a human.** A verb is run by an agent more often - than by a person, so no interactive prompt and no interactive shell — - either is a hang, not a UX choice. -- **Streamline before shipping.** No small helper functions or - rendering layers where a few inline lines read fine; consolidate. -- **Be honest about provenance.** Third-party material we adapt is - "inspired from" upstream, clearly marked, with its license named — - never passed off as verbatim, never left unattributed. -- **Leave working files alone.** Don't edit files that are fine just - because a change nearby made them look touchable. - -- **No dead code.** If nothing in the current layer calls it, it doesn't - land yet. `lc --help` advertises only verbs that work. -- **Port with intent.** Prior implementations (this repo's git history, - and the `redesign_prototype` branch of the sibling `lightcone-cli` - checkout) are references, not sources of truth. Neither is the spec by - itself: the spec plus this file's Recorded decisions is the current - design, and the decisions override the spec where they disagree. -- **Every layer ships tests.** See the per-layer test list in spec §11. - -## Architecture (target) - -A project is `pyproject.toml` + `uv.lock` + `.python-version` — **uv is -the only environment substrate**. Mode is *derived, not configured*: - -``` -direct mode (default) containerized mode -───────────────────── ────────────────── -.venv in the project tree [tool.lightcone.image] declared -no image, ever ⇒ content-addressed image is the -recipes run under Landlock/ execution world: driver, workers, - Seatbelt recipes, probes all run in-image - from the baked /opt/venv -``` - -Identity: `env_version = sha256(uv.lock ‖ .python-version ‖ canonical -install-settings ‖ canonical [tool.lightcone.image] ‖ Containerfile.extra -hash)`, recorded beside `definition_version` rather than folded into it. -Every output records what -enforcement it actually ran under (`hermeticity`). See spec §1–§3, §7. - -### Namespace contract - -`lightcone-cli` ships the `lightcone.*` namespace via PEP 420 implicit -namespace packages. **`src/lightcone/` must not contain an -`__init__.py`** — that would turn the namespace into a regular package -and break coexistence with future sibling distributions -(`lightcone-ui`, etc.). - -Any new `lightcone-*` package must: - -1. Use src-layout (`src/lightcone//…`). -2. Not create `src/lightcone/__init__.py`. -3. Ship only its own subpackage under `src/lightcone//`. - -## Repository Structure (current) - -``` -src/lightcone/ # namespace — NO __init__.py -├── _sandbox_exec.py # the Landlock shim — stdlib only, zero lightcone imports -├── app/ # the `lc view` app: a LabServerApp; `python -m`, never imported by the CLI -│ ├── __init__.py # LightconeApp — directories, URL, auth required of every route -│ ├── routes.py # an output's recorded run, astra's paper cache -│ ├── templates/index.html # the page, after JupyterLab's examples/app -│ └── static/ # built from ui/ — gitignored, packaged by the build hook -├── cli/ # the CLI only: flags, rendering, exit codes -│ ├── __init__.py # exposes main(), lazily -│ ├── commands.py # lc init, lc run, lc materialize, lc status, lc build, lc view -│ └── compute.py # lc compute resources, launch, status, down -└── engine/ - ├── __init__.py # docstring only - ├── project.py # what a project is: convergence, discovery, mode - ├── dataset.py # the git + git-annex seam: how a project stores - ├── identity.py # env_version, definition_version, the lock scan - ├── image.py # the system layer: declaration, Containerfile, tag — pure - ├── container.py # runtimes, the build, the archived image — impure - ├── crate.py # the publication view: the repo as an RO-Crate — pure - ├── assets.py # an output: its directory, its manifest, its state - ├── plan.py # the spec, read as a graph of tasks - ├── worker.py # making one output; also the `python -m` entry point - ├── materialize.py # the driver: dirty gate, Dask, the save/restore loop - ├── run.py # what `lc run` is: the probe + the uv hop - ├── serve.py # what `lc view` is: find this project's server or start one - ├── compute/ # explicit allocations and borrowed Dask clients - │ ├── __init__.py # Compute: catalog, resolve, launch, status, down; connect() - │ ├── model.py # the shared Pydantic models, the Provider protocol, PROVIDERS - │ ├── catalog.py # compute.yaml with local defaults and policy - │ ├── runtime.py # private files, TLS material, the scheduler config - │ ├── local.py # local provider: validated OS process identities - │ ├── local_runtime.py # the detached LocalCluster owner - │ ├── slurm.py # Slurm provider: native commands, JobName + Comment - │ ├── slurm_bootstrap.py # stock Dask Nanny per rank; rank zero hosts scheduler - │ └── output.py # recipe bytes through Dask events - ├── sandbox/ # the exec boundary - │ ├── __init__.py # the public surface (detect, run, scope, the types) - │ ├── model.py # Policy · Capability · Attestation · Backend protocol - │ ├── policy.py # what a probe and a recipe may touch - │ ├── boundary.py # detect() + run(): the mechanism-blind half - │ ├── landlock.py # Linux backend - │ ├── seatbelt.py # macOS backend - │ ├── oci.py # containerized backend: the mount table as mechanism - │ └── denial.py # the denial UX - └── templates/ # the scaffold's file content - ├── __init__.py # loader; a renderer only where there is a value to decide - └── files/*.tmpl # the templates themselves, as real files - -ui/ # the page (npm): astra-ui over the Contents API, built into app/static/ -evals/ # agentic eval seed: prompt.md + tasks// -tests/ # pytest — mirrors src/ -``` - -## Development Commands +## Commands ```bash -uv sync --group dev # pytest (+ pytest-cov), ruff, mypy, datalad -uv run pytest -uv run ruff check src/ tests/ # --fix to apply -uv run mypy src/ -uv build # wheel + sdist (CI runs this only to publish) -cd ui && npm ci && npm run build # the `lc view` front end, into src/lightcone/app/ +uv sync --group dev # pytest, ruff, mypy, datalad, playwright, roc-validator +uv run pytest # whole suite +uv run pytest tests/test_plan.py # one file +uv run pytest tests/test_plan.py::test_name # one test +uv run pytest -k "foreign_write" # by keyword +uv run ruff check src/ tests/ # --fix to apply +uv run mypy src/ # strict +cd ui && npm ci && npm run build # the `lc view` front end → src/lightcone/app/static/ +uv build # wheel + sdist; runs the npm build via hatch_build.py ``` -The front end is the one non-Python build. `uv build` runs it through -`hatch-jupyter-builder` (declared once, as the hook's own `dependencies`, -which hatchling hands every PEP 517 frontend) and refuses to package -without it; an editable `uv sync` only tries, so Python work never needs -Node — `lc view` does, and refuses without the bundle rather than serve -a blank page. - -Test, lint and type-check are the whole loop, and they are what +Test, lint and type-check are the whole loop and exactly what `.github/workflows/{tests,lint}.yml` run. There is deliberately no task -runner in between — the pre-rebuild `justfile` was 90 lines of wrappers -around them. The other workflows are `eval.yml` (the agentic eval, -on dispatch or the `run-eval` PR label; re-trigger by re-adding the -label) and `pypi-publish.yaml`. - -## Key Invariants (layer 1) - -**The CLI/engine split.** `cli/commands.py` owns flags, console rendering, -and exit codes — nothing else. Everything about *what a project is* lives -in `engine/project.py`: - -| Symbol | Role | -|---|---| -| `converge(dir, *, write)` | The whole scaffold operation | -| `ConvergenceReport` | `created` / `repaired` / `unchanged` / `blocked` / `warnings`, plus `.converged` and `.as_dict()` | -| `project_name(dir)` | PEP 503-ish name from the directory name | -| `ProjectError` | The one engine exception; `_EngineErrorGroup` in the CLI turns it into a clean `ClickException` | - -The engine never imports click and never prints. `converge(write=False)` -is check mode — the *same* decision path with side effects switched off, -which is what keeps `--check` honest rather than a second implementation. -Every item therefore routes through `_Converger.item`, `.file`, or -`.blocked`; **nothing writes or records outside that mechanism.** `.file` -takes a *thunk*, so check mode renders no template at all. - -Two report distinctions that matter: a **warning** is something -convergence can see but must not fix (advisory — never affects -`converged`); a **blocked** item is one convergence cannot complete, and it -does count, so a report can never claim a project is converged while -something it owns is absent. - -**There is no project discovery, by decision.** `lc init` is handed its -directory (defaulting to `.`); `lc run` assumes the current directory is -the project root — `project.current_project()` checks only that the -environment is there (`pyproject.toml`, `uv.lock`, `.venv`) and does not -require an `astra.yaml`, so any uv project can be probed. No walk-up: -the directory you invoke from is the project, or it is a clean error. - -**Two questions about a project root, not one** — the same shape as -`_in_repository` / `_can_ask_git` below. `declared_project()` wants only -what the repository carries (`pyproject.toml`, `uv.lock`); -`current_project()` adds `.venv`. The split is named rather than a -`synced=` flag on one function, because "what makes a directory a -project" should not be negotiable per call site, and because a slice of -a constant would make the answer depend on the order its entries happen -to be in. The weaker question has exactly one caller — the worker entry -point, which builds the `.venv` a moment later — and that is the whole -reason it exists. - -**CLI startup stays cheap.** `commands.py` imports the engine *inside* the -command callbacks and builds the rich console lazily, so `lc --help` and -shell completion pay for neither. Keep this up as verbs land: a module-scope -engine import would make every invocation pay for the heaviest layer. - -**The scaffold comes from `astra.scaffold`, not `astra.cli`.** Both export -`create_boilerplate`, and the second drags Click, Rich and the validation -stack — measured, 37 ms against 4 ms. `astra.scaffold` is stdlib-only -(checked: it pulls no linkml_runtime, click, rich, pydantic or -jsonschema), which is why this one astra import sits at module scope in -`project.py` where `astra.validation` and `astra.resolve` must not. - -**Templates are files, not string literals.** `engine/templates/files/*.tmpl` -are package data, loaded through `importlib.resources`. Placeholders are -`string.Template` (`${name}`) — **not** `str.format` — because several -templates legitimately contain braces (TOML tables, MyST `{astra}` roles). -Substitution is strict, so a missing key raises. - -**A template gets a function only when there is something to decide** — a -value the caller supplies (`pyproject`, `datalad_config`, `index_md`) or a -merge policy for a file the user already owns (`gitignore_repair`, -`gitattributes_repair`). Everything else is read by name: -`templates.read("myst.yml.tmpl")`, or `partial(templates.read, …)` where -convergence wants a thunk. This reverses the earlier "one function per -scaffolded file" rule, which had five of the module's twenty functions -doing nothing but rename a file — a second place for the name to be -wrong, and one the type checker cannot catch. The two `*_repair` -functions stay named because `_Converger.file` hands them the text alone, -so the template name has to be bound before the call site. - -**No engine constants for the environment — in direct mode.** The -scaffolded `.python-version` is the interpreter `lc` itself is running -on, and `requires-python` is that interpreter's minor as a floor. Both -come from one place, so they can't conflict, and neither is a number to -maintain. Identity follows the project's files from there: a direct -project's `env_version` hashes `.python-version`'s bytes, not anything -in the engine. The scoping is deliberate: containerized mode's default -base and uv digests *are* engine constants, by spec-§2 design — see the -layer-6 invariants for what moves when they do. - -**`.gitignore` and `.gitattributes` converge entry-wise, not by marker.** -`templates.entries(name)` is a template minus comments and blanks; -`templates.missing(name, text)` is what a repair appends, in template -order. Idempotency is therefore structural — a pattern already in the file -is never re-added, whoever wrote it — and a pattern introduced by a later -lc release still reaches projects that already have a `.gitignore`, which -a "marker present ⇒ done" check would have skipped. The header comment -(`templates.header`) is cosmetic only; never make correctness depend on -it. - -**What `lc init` converges** — idempotently, never overwriting a file the -user owns: - -| Path | Role | -|---|---| -| `astra.yaml` + `universes/baseline.yaml` | astra's scaffold, verbatim: an **empty** analysis (`inputs: []`, `outputs: []`, `decisions: {}` — the root collections are required fields) and a baseline that selects nothing, which exists because `plan.build` refuses a project with no universe. **One item keyed on `astra.yaml`**, so the baseline never lands beside a user-authored spec | -| `pyproject.toml` | The uv project: **virtual** (no `[build-system]`), no dependencies — the engine is the host's uv tool, never a project dependency (see Recorded decisions), so the lock carries only what the analysis imports | -| `.python-version` | The exact patch of the interpreter `lc` is running on | -| `uv.lock`, `.venv` | **Derived** — converged by correctness, not existence: `uv lock --check` / `uv sync --locked --exact --check` decide, then `uv lock` / `uv sync --locked --exact --compile-bytecode` repair | -| `.gitignore` | One managed block of patterns; convergence ensures each is present | -| `.git` + the annex | `git init` then `git annex init` — results are versioned in the project's own repository — then **`filter.annex.required=true`**, the one key git-annex does not set and the difference between a loud refusal and silent corruption (see the storage invariants) | -| `.gitattributes` | The storage policy: what git-annex holds and what git carries. Line-managed, like `.gitignore` | -| `.datalad/config` | A `datalad.dataset.id` UUID, generated once. Read back only by `dataset.dataset_id`, through `git config -f`, for the run record's `dsid` | -| `data/` + `README.md` | Where declared inputs live; annexed, and committed before anything computes on them | -| `results/` + `README.md` | Where outputs land; the README states the materialize-don't-hand-write contract | -| `myst.yml`, `index.md` | Template MyST report referencing `astra.yaml` *by path* | - -- **Only what git can carry is converged.** No `src/`, and no empty - `universes/`: git does not track empty directories, so converging one - reports drift on every fresh clone, forever. astra dropped `src/` for the - same reason (astra-tools#100) — where analysis code lives is the user's - layout. - Universes are discovered by `glob("*.yaml")`, which is empty-not-error on - a missing directory. `tests/test_project.py::test_a_clone_of_a_converged_project_is_converged` - pins this: a clone must need nothing but `.venv` and `git annex init`. - Those (with the annex filter, a `.git/config` entry) are the - exemptions, and for one reason — they are local state git does not - clone. -- **Convergence, not scaffolding.** Each item is created if missing, - offered to a conservative `repair(text) -> str | None` hook otherwise, - and left alone when the hook returns `None`. `--check` computes the - same report without writing (exit 1 when not converged); `--json` - emits `{converged, created, repaired, unchanged, blocked, warnings}`. -- **Derived artifacts converge by correctness, not existence.** `uv.lock` - and `.venv` go through `_Converger.item`'s optional `is_current=` - predicate, which is uv's own no-write verification (`uv lock --check`, - `uv sync --locked --exact --check` — both confirmed read-only against uv - 0.12.3). A lock that no - longer matches `pyproject.toml`, or an environment that no longer matches - the lock, is exactly as unconverged as a missing one and reports as - `repaired`. Existence alone made `converge()` a no-op on drift, and - everything that converges before acting — `lc materialize`'s sync, the - worker entry point's — would have silently inherited it. - - The probe is skipped when the artifact is absent (nothing to ask), so a - fresh project costs none and the created/repaired split falls out of the - same check. - - Check mode may *probe* but never mutates; `test_check_mode_only_probes` - pins that every uv call it makes carries `--check`. - - **Honest limitation** (measured, uv 0.12.3): `uv sync --check` catches - packages the lock requires and the environment lacks, but not *extras* — - a hand-installed package leaves it reporting "would make no changes". - Set-level, not byte-level, as spec §3 already accepts; what bounds what - a recipe can import is the sandbox, not this probe. - - **The probes are why the scaffold sets a uv floor** (`required-version - = ">=0.7"`, 2026-10). uv 0.6 has no `uv sync --check` and 0.5 no - `uv lock --check`; both reject the flag, `_is_current` reads that as - "not current", and every convergence silently re-locks and re-syncs - forever. 0.7 is the oldest release where every flag lc runs exists - (measured back to 0.5: init, `--check` and the recipe hop), so the - floor names a requirement, not the uv the design happened to be - measured on — the earlier `>=0.12` was that, and refused `lc init` - on every host lagging it. The `uv-floor` job in `tests.yml` runs the - uv-touching suites under exactly that version, so a flag lc starts - using that the floor lacks fails CI instead of silently. Raise both - together. Accepted residue: uv 0.12 changed the default pre-release - mode, so collaborators either side of it can resolve a pre-release - differently and rewrite each other's lock — churn (`behind`), never - a correctness loss, since the lock is recorded. Projects scaffolded - with `>=0.12` keep it: `pyproject.toml` is the user's. -- **An authored `Containerfile` is left alone.** Spec §8 has `lc init` - refuse one; we don't. Images are generated from the lock, so a - hand-written file is simply not an input to anything — treating it as a - blocking half-state was more ceremony than the situation earns. Revisit - if the container hatch (layer 6) turns out to need the disambiguation. -- **Nothing about uv's caching or linking is overridden**, because uv's - defaults are already right: it clones (copy-on-write) or hard links - package content out of a global cache, so projects share it. Measured - here: two environments cost 284 MB together rather than 432 MB. The docs - discourage forcing `symlink` mode (it couples every environment to the - cache's survival), and **`--system-site-packages` must never be used** — - it would make packages outside the lock importable, which is exactly what - the environment model exists to prevent (spec §7, G6). - - The sharing silently stops working when the cache and the project are - on **different filesystems** (uv falls back to full copies). uv warns; - `tool_warnings()` lifts that warning out of uv's progress output into - the report, so it reaches both the console and `--json`. - - At NERSC, `UV_CACHE_DIR` must move off `$HOME` for a different reason: - compute nodes cannot lock files there (uv fails with os error 524), - and every recipe's `uv run` hop locks the cache. The user exports it - before `lc compute launch`, which the Slurm job inherits; - `child_env` keeps it (`_UV_KEPT`). Documented in the user guide rather - than configured, by decision (2026-09). - - `--compile-bytecode` is the one genuinely per-project cost: bytecode is - generated into the venv, never linked (~55 MB of 216 MB here). It is a - deliberate trade: paying compilation once here beats paying it on the - first import of every run. -- **Every external tool goes through one seam**, `project._run`, which - tests monkeypatch — so the suite never shells out, and every call is - inspectable. `_check_call` turns a nonzero exit into a `ProjectError`: - nothing convergence invokes is allowed to fail silently. Every uv - invocation carries an explicit `--project` — uv's own walk-up discovery - is never trusted (spec §4). -- **uv, git and git-annex are all required.** Each is a refusal, not a - warning: uv is the environment substrate and git + git-annex are the - storage substrate, and results are versioned in the repository, so there - is no useful project without any of them. git is the one tool uv cannot - install and the single admitted exception to a uv-installable stack; - git-annex ships as a wheel and is therefore a dependency of the lc tool - itself, which makes its wheel platforms the CLI's install floor. - (This reverses layer 1's original "git is optional" — a project without - version control had nowhere to put a result.) -- **Two questions about git, not one.** `_in_repository` is a pure - filesystem walk-up and answers for a directory that does not exist yet, - which is what check mode needs. `_can_ask_git` adds `is_dir()`, because - every git *invocation* needs an existing working directory — inside an - enclosing repository the walk-up says "in a repository" for a - directory that is not there, and running git in it raises - `FileNotFoundError` out of `Popen` rather than answering anything. -- **`git init` checks for an *enclosing* work tree**, not just a `.git` in - the directory, so `lc init subdir/` inside a repository can't create a - nested one. (`.git` may be a file — linked worktree or submodule — so the - test is `exists`, not `is_dir`.) The annex asks git-annex's own question, - `git config --get annex.uuid`, so an enclosing repository that already - has one is adopted rather than re-initialized. -- **`lc init` has exactly two flags**, `--check` and `--json`. `--no-git` - and `--no-sync` were deleted: neither had a caller outside the test suite, - `--no-git` was a workaround for the missing enclosing-repo check, and - `--no-sync`'s real home is containerized mode (layer 6), where the host - `.venv` is inert. Don't add a flag whose only user is a test — stub - `project._run` instead. -## Key Invariants (storage) - -**Results are versioned in the project's own repository**, on the DataLad -model: **git carries the pointers and the history, git-annex carries the -bytes.** `engine/dataset.py` is the whole seam, and every command in it -goes through `project._run` — the same one convergence uses, so there is -one monkeypatch point and the `tools` fixture already covers git. - -**git-annex is a wheel, and that sets the CLI's install floor.** -`manylinux_2_34` on x86_64/aarch64, macOS 14+ arm64 or 15+ x86_64, -win_amd64 — and **no sdist**, so a host below the floor fails to install -rather than building from source. Because git-annex is a *hard* runtime -dependency, that floor gates installing the lc tool at all, including -`lc run`, which never touches the annex — but only the tool: projects no -longer depend on lightcone-cli, so the floor never constrains a project's -own resolution. If it ever bites a real user, the hard dependency is the -thing to revisit — an extra, or a probed requirement like git — not the -floor. - -**Perlmutter clears it** (checked 2026-08-19, login node): the wheel -installs and `git annex version` runs. That was the open question this -stack was most likely to fail on, so it is written down rather than -re-derived. Two things it does *not* settle, both layer 7's: - -- **Lustre/GPFS behaviour is unmeasured.** arXiv:2505.06558 documents - symlink, many-small-files and inode pressure on parallel filesystems. - Correctness is not the worry; cost is, and one output is one file plus - one hard link to its object. Results are already thin (above); adjusted - (unlocked) branches remain the untried lever, as does whether Lustre - makes the hard link behave differently. Time `git annex add` and - `git annex get` on `$SCRATCH` and `$CFS` before designing around - either. -- **`git-annex-shell` is not on the default remote PATH.** `git annex get` - from a laptop dispatches to it over a non-login ssh session, and the - wheel installs it beside the tool's interpreter. Configuration, not - design: `git config remote..annex-shell `. - -**`filter=annex` is what makes an ordinary `git add` do the right thing.** -`git annex init` configures the smudge/clean filter itself; the template -adds `* filter=annex`, and `annex.largefiles` then decides what counts as -content. So a researcher types `git add -A . && git commit` — the same -git they already know — and a 200 KB input lands in the annex as a -101-byte pointer while `src/main.py` and the manifests stay real. Nothing -lc scaffolds, prints or documents asks anyone to run a git-annex command, -and `dataset.save` does not run one either. - -**The `annex.largefiles=nothing` default is load-bearing.** It has to come -first, with outputs and inputs opting out; last matching line wins. -Without it `filter=annex` routes *everything* into the annex, analysis -code included. `tests/test_dataset.py::test_analysis_code_stays_in_git_and_stays_writable` -pins it against a real annex. - -**Manifests stay in git, deliberately.** `results/**/.*.manifest.json` -is exempted back out of the annex so it is readable on a clone that has -fetched no annex content at all — which is what lets `lc materialize ---check` classify a whole project on a laptop that holds none of the -bytes. - -**An unfetched file exists, and that is the trap — in two shapes.** -`assets.data_version` refuses both with `ContentNotFetchedError`, naming -`git annex get`, and it checks for both regardless of which one lc's own -writes produce: `annex.thin` and `git annex lock` are the researcher's to -set on their clone, so the shape a file arrives in is not ours to assume. - -- **Unlocked** — what `filter=annex` writes, hard-linked or copied alike. - A clone without content holds a ~100-byte *pointer file* where the data - would be: it exists, it is readable, and hashing it yields a perfectly - well-formed digest of the wrong thing. The test follows git-annex's own - `isPointerFile`: a file no larger than 32 KiB whose bytes begin - `/annex/objects/`. Measured before it was fixed: the same input hashed - differently on a clone, silently. -- **Locked** — a symlink into the object store, which without the content - dangles. This one is quieter and worse: `is_file()` answers False for a - dangling symlink, so a directory walk filtered on that alone drops the - absent file from the digest *without a word*, reporting a hash of - whatever subset happens to be present. Only dangling symlinks are added - back to the walk — one that resolves to a file already is one, and one - that resolves to a directory is not content. - -`--check` catches the error and reports "not in this clone" rather than -"changed", because the two are different facts and only one of them means -a rebuild. - -**git-annex is on `PATH` by construction, not by runtime repair.** -`git annex` is not a builtin — git dispatches it by searching `PATH` for -a `git-annex` executable — and an installer links only the requested -package's own executables, so a plain `uv tool install lightcone-cli` -would have left the one git command we tell people to type unable to run. -The fix is metadata: lightcone-cli re-declares the git-annex wheel's four -entry points verbatim in its own `[project.scripts]` (`git-annex`, -`git-annex-shell`, `git-remote-annex`, `git-remote-tor-annex`, all -`git_annex:cli`). The wheel ships no raw binaries in `bin/` — its -executables *are* that dispatcher, which `execv`s the real binary out of -package data with `argv[0]` preserved, and git-annex dispatches on -`argv[0]` busybox-style. So every install channel carries the -executables wherever it carries `lc`: the tool install links all five -names (verified), `uv run` and `uvx` front the venv's bin, and the -rerun's ephemeral engine environment gets them the same way. That covers -lc's own subprocesses *and* the researcher's bare `git add` with one -mechanism, which is why the old `put_our_bin_first()` PATH-prepend was -deleted rather than kept as a belt: a second answer to "which annex runs" -that could disagree with the shell's. - -The declaration is mirrored, never invented: -`test_the_annex_executables_are_ours_to_install` asserts ours match the -wheel's exactly, so an executable upstream adds, drops or renames fails -the suite rather than every user's install. (Considered and rejected: -there is no pyproject metadata to link a *dependency's* executables; -`--with-executables-from` is an install-time flag users won't type; a -post-install link repair contradicts "the install command is enough"; -vendoring the binaries into platform-specific lightcone-cli wheels is -five 14–36 MB wheels and a repack pipeline for what four lines of -metadata provide.) - -The accepted residue of dropping the prepend: `PATH` order is the -user's, so a system git-annex fronting the install's wins the dispatch — -for lc's subprocesses exactly as for their own git, and no different -from which `git` itself runs. Nothing records the annex version anyway -(see the Recorded decision on the engine's dependency closure). - -**"By construction" covers every install channel — except the -researcher's own shell**, which is the one place `filter=annex` sends -git looking for git-annex (the filter config `git annex init` writes is -`PATH`-resolved). Under `uvx` nothing lands on the user's `PATH`, and a -`git add` whose clean filter cannot start prints an error, **exits 0, -and stages the raw bytes into git history** (measured, and pinned by -`test_stock_plumbing_without_required_stages_raw_bytes_silently`). So -dataset convergence sets one key as an item (`annex-filter`, 2026-08): -**`filter.annex.required=true`, always** — git's own hard failure -instead of silent corruption. - -**That flag is the whole of it, by decision.** lc writes no absolute -path into a repository and rewrites no filter driver or hook: how git -dispatches git-annex stays `PATH` resolution, which is git-annex's own -design and git-lfs's too (it writes `filter.lfs.required = true` beside -`PATH`-relative drivers and has never baked a path). - -**The hazard is Linux's, the guarantee is ours.** The exits-0-and-stages-raw-bytes -behaviour is what Linux git does; macOS git aborts the same handshake -outright (exit 128, `fatal: the remote end hung up unexpectedly`) and -would have been safe without the flag — found by CI, which is why the -suite runs on both. So `test_stock_plumbing_without_required_stages_raw_bytes_silently` -is Linux-only by `skipif`, and the loud-refusal test asserts the facts -(nonzero exit, nothing staged) rather than either platform's wording. -Do not read macOS's accident as a reason to scope the flag: an -implementation detail that can change on the next git release is not a -guarantee, and Linux is where the venues are. - -The rule the neighbours agree on is **route through a filter ⇒ set -`required`**, and both halves were measured. git-lfs routes and sets -it. `datalad create` does *neither*: no `required`, and no -`* filter=annex` in its `.gitattributes` (just `* annex.backend=MD5E` -and `**/.git* annex.largefiles=nothing`) — so a plain `git add big.bin` -in a DataLad dataset stages 200 000 raw bytes into git *even with -git-annex working*, and the safeguard is procedural: use `datalad -save`. lc made the opposite promise — the researcher types the git they -already know — so lc owns the failure mode that promise creates. The -honest cost, also measured: once a project holds committed annexed -content, the refusal covers `git status`, `git diff` and `git checkout` -too, not just `git add`. Pinning the filter -and the four hooks to the engine's bundled executable was implemented -and reverted before merge: it writes durable state pointing at a -prunable `uvx` cache, `core.hooksPath` can send the hooks into a -machine-shared directory that breaks commits in unrelated -repositories, and uv documents `uvx` environments as disposable. The -reachability problem is the *install's* to solve, not the -repository's — `uv tool install lightcone-cli` links the git-annex -wheel's entry points beside `lc` (verified), so `ambient` git-annex is -on `PATH` for free and the whole question disappears. `uvx` is fine for -running lc and cannot support the researcher's bare `git add`; that is -what the docs say, and reporting it from -`lc init`/`lc status` is an open follow-up, not a promise made here. -`require_git_annex` stays a `PATH` check, deliberately — it gates lc's -*own* `git annex` subprocesses, which dispatch from lc's environment, -where every install channel fronts the bundled copy. - -**A project can sit inside a larger repository, so `dataset.status` is -scoped and relativised.** `lc init subdir/` adopts an enclosing work -tree rather than nesting a new one — that is a supported layout — and -`git status --porcelain` otherwise covers the *whole* work tree and names -paths from *its* root. Unscoped, an edit anywhere else in the repository -refuses every run in the project, and lc's own writes arrive as -`subdir/results/…`, which the refusal's path-class split cannot -recognise. So the call carries `-- .` and the `rev-parse --show-prefix` -is stripped off each path. A wholly untracked project collapses to `.`, -which is git's own summary of it and exactly what the refusal should tell -someone to add. - -**git needs an identity, and that is asked before a run, not at the first -commit.** A fresh container or CI image has none — the case an agent-run -CLI meets most — and discovering it at `dataset.save` would throw away -whatever the recipe had already computed. `require_committer` asks -`git var GIT_COMMITTER_IDENT`, which is the question a commit itself -asks: an identity resolves from `user.email`, `EMAIL`, the author and -committer variables, or three levels of config, and a probe that -reimplements that lookup is one that can disagree with the thing it -stands in for. Measured: `git annex init` does *not* need one (it -tolerates the missing identity and exits 0), so `lc init` is not gated on -it — only the verb that commits is. - -**The ignore probe asks about the directory as a directory.** -`check-ignore` is run with a trailing slash (`results/`), because the rule -that matters most — the `results/*` an older lc scaffold wrote — ignores -the directory's *contents* and does not match the bare name at all. And -with `--no-index`, which asks about the *rules* rather than the index: -without it git answers "not ignored" for anything already tracked, which -is exactly the project where someone committed one result by hand and left -the rule for the next. The item records **`blocked`**, not a warning — -`git add` skips ignored paths in silence, so a materialize would report -success and commit nothing — and it names `::`, -because `.gitignore` convergence only ever appends and cannot fix this. - -**A new managed `.gitattributes` line is always appended at the *end* of -the template.** Repair is append-only, so a project scaffolded under an -older template receives new lines at the bottom of its file — and -`gitattributes_disorder` judges against template order. A line added -anywhere but last makes lc write an order it then reports as blocked, on -every project scaffolded before the line existed. The layer-6 image line -sits after the manifest exemption for exactly this reason (the patterns -are disjoint, so the order is free), and every future line takes the -same slot. - -**`.gitattributes` can be *unrepairable*, and that is a second blocked -item.** The file is last-match-wins, and a repair only appends — so a -project that already carries `results/** annex.largefiles=anything` -without the `*` defaults gets `* annex.largefiles=nothing` appended -*below* it, and every result then lands in git as a plain blob while the -report says repaired and converged. `templates.gitattributes_disorder` -judges **the text a repair would produce**, not the text as it stands (a -file missing the defaults is in order until they are appended), and only -compares lines setting the **same attribute** — `* filter=annex` landing -under an `annex.largefiles` line is not disorder, and treating it as one -blocks a perfectly good file. Blocked rather than fixed, for the reason -the ignore rule is: reordering a file the user wrote is not something -append-only convergence gets to do, so it names the line and prints the -order the managed lines belong in. - -**An lc project is a DataLad dataset from birth.** `.datalad/config` -carries a `datalad.dataset.id` UUID, generated once by `lc init` and never -regenerated — it identifies the dataset across clones and siblings. -Verified, not assumed: `datalad status` recognises a freshly scaffolded -project with no `--force` adoption step, and `Dataset('.').id` is the UUID -we wrote. The reciprocal is a standing non-goal: **lc never requires -datalad, never imports it, and never parses `.datalad/`.** lc *writes* -under it — the dataset id, and since layer 6 the image archives and the -`datalad.containers.*` config keys — but reads nothing back except -through `git config -f`, and the archive is read as an image, never as -datalad state. A researcher who wants `datalad get`, siblings or RIA -stores runs `uv add datalad` in their own project. - -**Annexed files are ordinary writable files.** `filter=annex` keeps them -unlocked, so an output can be overwritten in place and nothing needs to -remove it first. (This reverses the earlier symlink model, where a -rebuild hit `PermissionError` on a path that looked perfectly ordinary.) - -**Results are committed thin; declared inputs are not.** `dataset.save` -passes `-c annex.thin=true` to its `git add`, so a result is hard-linked -to its annex object instead of copied — measured, 39 MB to 20 MB for one -20 MB file, since an unlocked file otherwise exists both in the tree and -in the object store. - -It is safe *there* and nowhere else, and the reason is specific: -**thin's hazard is an in-place write.** With the file hard-linked to the -object, one `open('r+b')` rewrites the object under the key that names -it — measured, the committed version becomes unrecoverable (`git annex -get` reports no known copies) and `fsck` only notices later, moving it to -`.git/annex/bad/`. lc never writes in place: a worker *removes* an output -directory before rebuilding it, and an unlink merely drops the link -count, leaving the object intact (measured — an old version still checks -out clean afterwards). So the flag is passed **per-add and never written -to the repository's config**, because repo-wide it would reach `data/`, -which researchers add with their own `git add` and whose tools — -`h5py.File(p, 'r+')`, astropy `mode='update'` — very much do open files -for update. - -Thin-ness is *not* recorded anywhere: it is local working-tree state, so -a clone re-decides at `git annex get` time. That is why detection has to -handle every shape rather than the one we write. - -**`restore` is scoped, and asymmetric.** `git clean -qfdx` always, plus -`git checkout HEAD --` **only when HEAD has the path** — a first -materialization has nothing to go back to, and the naive form exits -nonzero on the pathspec. Never `git checkout HEAD -- .`: a failed task -must not discard edits made elsewhere while the graph was running. - -## Key Invariants (layer 2) - -**Two hashes, and they answer different questions** (`identity.py`). -`definition_version = sha256(recipe ‖ canonical decisions ‖ format)` is -what the spec says an output *is*. The format is framed in its own right -rather than left to arrive through the rendered `{output}` path: a recipe -need not use that placeholder at all (`evals/tasks/snae` does not), and -one that does not would keep its digest across a re-declared -serialization — while the sidecar, named from the id alone, does not move -either, so the manifest would go on describing an output at a path that -no longer exists. `env_version = sha256(uv.lock bytes ‖ -.python-version bytes ‖ canonical install-settings JSON)` is what it ran -under. - -**`env_version` is not part of `definition_version`, and that is the whole -shape of the model.** An environment moves for reasons that have nothing -to do with any particular output — one `uv add` for one plotting script -rewrites the lock for the entire project — while a research artifact costs -hours to remake and has usually already been looked at. So an environment -edit stales nothing; it makes an output **behind**, which is reported and -left alone. (This reverses the original design, where `env_version` sat -inside `code_version` and therefore staled every output in the project on -any dependency change, and every output in every project on an lc upgrade. -That was recorded as an accepted cost; it was the bug.) - -Nothing is lost by not rebuilding: the manifest records the environment -*and* the commit, and that commit's `uv.lock` reconstructs the project -environment exactly. (The engine is not in that lock — its reconstruction -is the run record's job, and its version the manifest's `lc_version`.) -Over-sensitivity in `env_version` is affordable precisely because it no -longer spends compute. - -**Both are length-framed.** Concatenating fields raw lets a boundary shift -between them produce one digest from two different inputs. `_frame` -writes label, length, then bytes. The test lives on `env_version`, where a -shift is actually constructible — two raw file bodies, adjacent — rather -than on `definition_version`, whose second field is canonical JSON and -cannot be shifted into. Mutation-checked: breaking `_frame` fails it. - -**The lock's raw bytes, not a parse.** A comment reflow moves -`env_version`, deliberately: the alternative is a parse of our own that -can silently disagree with uv about what the lock means, and -over-invalidation is the failure that costs time rather than correctness. - -**The install-settings list is closed** (`_INSTALL_SETTINGS`), and every -key is hashed whether or not the project sets it. A setting outside the -list must not move the hash, or every uv config nicety stales the world; -a setting whose value merely *matches* today's default must, because that -default can change under a project that never said anything. - -**The settings are read where uv reads them, and `uv.toml` *replaces* -`[tool.uv]`** rather than merging with it (measured, uv 0.12.5 — uv warns -about the pair itself, and `tool_warnings()` already lifts that into the -report). Reading both would hash settings uv is ignoring, reporting two -environments where uv installs one; reading only `pyproject.toml` left a -`uv.toml` free to change what gets installed without moving `env_version` -at all. Only the *values* are hashed, never which file supplied them — -two projects that install the same artifacts are one environment however -they spell it. `scan_lock` reads `default-groups` through the same -function, because it is asking uv's question too. - -What this deliberately cannot reach is **machine-level configuration** -(user `~/.config/uv/uv.toml` and system `/etc/uv/uv.toml`), which uv -merges in underneath the project's own (measured). That is machine -state, not project state: hashing it would -make one commit answer differently on two hosts, so a colleague's clone -would report every output as behind. **There is no flag that closes -it**: `--config-file` refuses a -`pyproject.toml` outright, and both `--config-file ` and -`--no-config` drop the project's own `[tool.uv]` along with everything -else (measured; `--no-config` also drops `[tool.uv.index]`, which is the -GPU mechanism, so adopting it would break working projects). Since the -hardening pass the hole is *annotated at run time* (issue #176's -advisory option): `identity._machine_config` checks the two documented -paths per platform — those levels can only ever be a `uv.toml`, so the -probe is complete, and it tests which *keys* a file sets because list -settings concatenate across levels — and a hit lands in `scan_lock`'s -advisory tier beside `sdist_built`. Never hashed, still. The env-var -spelling of the same hole (`UV_NO_BINARY` and friends) is *closed*, not -annotated: `project.child_env` scrubs ambient `UV_*` outside a plumbing -allowlist (`_UV_KEPT` — cache dir, link mode, the managed-interpreter -store and its mirror, timeouts, TLS, air-gap, credentials, uv's own -recursion guard), and every uv-acting verb names the non-empty -variables dropped through `project.uv_scrub_warning` — one composer, -one predicate with the scrub: convergence puts it in the report (so -`lc init` says it), materialize in its warnings, and the probe in its -outcome's notes, which is why the CLI never composes it (issue #179). -`lc build` deliberately says nothing: the image's uv runs inside the -container build, where the host environment does not reach. The suite -blinds itself to the host's machine config via -the autouse `machine_uv_config` fixture — `/etc/uv/uv.toml` has no -environment variable to scrub. - -**The git commit is recorded, never hashed, and never a signal.** It goes -in the manifest so the code that produced a result stays recoverable. It -is out of `definition_version` because git has one sha for the whole -tree — hashing it would stale every output in the repository on a README -edit — and it is not a `behind` trigger for the same reason: it moves on -every commit, and a signal that is always on is not one. - -The honest consequence, which is a **decision and not an oversight**: -editing `src/fit.py` remakes nothing, because the recipe *string* is -unchanged. Per-output code invalidation is available by declaring the -source files as ASTRA inputs, and that declaration is deliberately the -researcher's to make rather than something lc infers — for an expensive -output, "the code moved and the result still stands" is a legitimate and -common position. Do not add a heuristic that scans a recipe's command line -for repo paths. - -**Names are compared in PEP 503 form.** uv writes the normalized name -into `uv.lock`; `pyproject.toml` carries whatever the author wrote, and -`project_name()` keeps `_` and `.`. Raw comparison makes a packaged -project called `my_project` fail to recognise *itself*, and the lock scan -then refuses the whole run over the project's own code. - -**The lock scan refuses only what cannot be audited.** A path, directory, -or editable dependency records *where* it was rather than *what was in -it*, so two syncs of one lock can install different code while every hash -agrees they are identical — that is a refusal. A registry package with no -wheel is a *report* (identity covers the sdist, not the build of it), and -a non-default dependency group is *advisory*. The project's own package -is exempt: it is the project, and the repository already records its -bytes. - -## Key Invariants (layer 4) - -**What the spec *means* is ASTRA's to say, and `plan.py` asks rather than -re-derives it.** `astra.resolve` settles each universe's decisions, -resolves every output's inputs to what supplies them, drops the outputs -whose `when:` does not hold, and renders the recipe grammar. Scoping, -`from:` aliases, sub-analysis nesting and the placeholder grammar are all -*read* here, never re-implemented — a second implementation of one -specification is how the two start disagreeing, and ours had: - -- it could not build `examples/iris_pipeline` at all, ASTRA's own - canonical nested example; -- it ignored `when:` on an output, so a universe ran a recipe the spec - excludes and committed a manifest for it; -- it invented a dotted input id (`inputs: [hod.mass_function]`) and an - implicit "same universe id" fallback for sub-analyses, neither of which - `astra validate` accepts — `UniverseNode.universe` names it explicitly. - -`lightcone.engine.plan` therefore holds only what execution adds: -`Task`, `Graph`, and the mapping of resolved outputs onto directories, -edges and `definition_version`. When something about the spec's meaning looks -wrong, the fix is in astra-tools, not here. - -**A spec ASTRA rejects never reaches a recipe.** `build` runs -`validate_analysis_schema`, `validate_analysis_file` and -`validate_universe_file` before resolving anything, and refuses with -ASTRA's own errors. This is a contract requirement, not a courtesy: -resolution answers what a *valid* spec means and does not re-check that -it is one, so without the gate an invalid spec surfaces later as a -missing decision or an unresolvable input — blaming the run for a fault -in the file, far from the line at fault. It caught three of lc's own -test fixtures the first time it ran. - -**One output is one file, and the spec names it.** An output is -`results//.` — never a directory a recipe -fills — and the path in a rendered recipe *is* the path on disk: no -staging, no scratch, no relocation. `format` comes from ASTRA -(`Output.format`, astra-spec 0.0.14), so lc composes the filename rather -than a recipe choosing it, and the whole contents of `results/` are a -pure function of the spec. `plan.build` **refuses** a spec whose -executable outputs omit it, naming every one: it is only *recommended* -until ASTRA 0.1.0, but lc has nowhere to write an output without it. - -**One rule names a path, and both the recipe and the run record use it** -(`plan.declared_path`). Project-relative inside the tree, absolute -outside it, never resolved. A declared input may name an absolute -`source:` — ASTRA allows it and an HPC project pointing at a shared -catalog is the obvious case — and there is no project-relative spelling -for one, so a bare `relative_to` was a `ValueError` traceback out of -`lc status`, `--check` and `materialize` alike. Two copies of this rule -existed; the second is what made the first easy to miss. - -An input outside the project is **reported, not refused**: its bytes are -hashed into the manifest like any other, so a change to it still -cascades, but it is not in the repository and the commit recording the -output cannot bring it back. That is a weaker promise than the rest of -the layer makes, and saying so is the whole obligation — the same -treatment `sdist_built` gets. - -**Two universes cannot share an id.** The id names a directory under -a `results/`, and the graph is keyed on `(universe_id, output_id)` — so the -second file simply replaced the first and one universe's outputs went -missing with nothing said. The way in is the natural one: copy -`baseline.yaml`, edit the decisions, forget the id inside. `build` -refuses, naming both files. - -**Because the path is composed, `assets.output_path` refuses any part -that is not one path component** — an empty or `..`-bearing universe or -output id would place a file outside the tree every guard above it -checked — and a `format` that could not be an extension (empty, a -separator, or leading-dot, which would make the output look like its own -sidecar). It also refuses an output id carrying a **dot**: the sidecar is -recovered by partitioning the filename on the first one, so a dotted id -would name a manifest for something else. ASTRA's own id grammar carries -no dot, so this only ever fires on an id lc did not compose. - -**The reset takes what the output's id names, never the directory.** -Outputs share a directory now and Dask writes them concurrently, so a -whole-directory delete would take a neighbour's bytes. It unlinks the -sidecar and globs `.*`: an id cannot contain a dot, so the glob -cannot reach a sibling, a longer id, another output's sidecar or a scope -directory of the same name — and it *does* reach a payload left by a run -that declared another `format`, which is what stops one orphaning. - -**A payload that is not a regular file fails the task.** `data_version` -branches on `is_file()` and hashes a directory happily, so `mkdir -{output}` — what every recipe written for the old layout does — would -otherwise commit a well-formed digest of something that is not the -output, plus a crate `File` with no `sha256`. One check covers absent, -directory and wrong-name, and it is what makes exit 0 stop being -evidence that anything was written. - -**The manifest is a sidecar, `..manifest.json`, named from the -id alone and never the format.** So it keeps its path — and therefore its -history — across a re-declared serialization, which is what lets -`_foreign_write` still answer for an output whose payload path is new. -Decomposition is safe by partitioning on the **first** dot (ids carry -none; formats may, `tar.gz`), never `Path.stem`. The old -`_HASH_EXCLUDE` is gone with the directory that made it necessary: the -manifest cannot be inside the thing it describes any more. - -**Dask owns the ordering.** Tasks that may execute receive upstream futures -or already-current `TaskResult` values as arguments, so dependency order, -parallelism, and scheduling fall out of the argument graph. There is no ready-set -loop and no hand-rolled topological sort in the execution path. -`Graph.order()` exists for the read-only walk, which has to classify a -task after everything upstream of it — and for submitting in an order -where a task's upstream handles already exist. - -**The driver owns git, alone.** Workers execute and return a -`TaskResult`; the driver commits, in one thread, as results arrive. -Concurrent git operations on one repository race on the index lock — -this is the `datalad-slurm` schedule/finish split, and it is not a -preference. - -**A dependent does run while its upstream is being annexed, and that is -safe.** Dask releases a task the moment its upstream's *worker* returns, -milliseconds before the driver finishes committing that upstream's -directory — so a recipe reads an input directory while git's clean filter -is moving its content into the annex. Measured before relying on it: git-annex -hard-links the content into the object store and then renames the symlink -over the file, so the path never stops existing and never holds partial -bytes (448 concurrent full-content reads across 24 MB: no missing paths, -no short reads, no wrong bytes). Don't "fix" this by moving the save into -the task — that is what puts git back in the workers. - -**A declared input is hashed once per run** (`assets.Versions`). Without -it a multiverse spec re-reads the same bytes once per `(universe, -output)` that names it — eight universes times four outputs sharing one -catalog is thirty-two full hashes of one file, paid again on the -"nothing to do" path because classification needs the digest before it can -skip. Memoizing is sound for exactly as long as a run lasts: a run -refuses to start on a dirty tree, and the only in-tree path a recipe may -write is its own output directory. A class rather than a closure, so it -keeps one dict alive and not whatever scope built it; and deliberately -unlocked, because a lock would serialise every hash and would not survive -being handed to a worker in another process. - -**HEAD is read once per run, by the driver, and handed down.** The driver -commits each output as it lands, so HEAD *moves* during a run: a -per-task `dataset.head` would stamp later manifests with a commit this -same run created, and whether it did would depend on whether a recipe -finished before or after the previous save. Nondeterminism in a -provenance field is worse than either answer. - -**The worker never raises, and that is enforced at the unit boundary.** -It returns `ok`, `current`, `behind`, `failed`, or `blocked`. A task whose upstream -did not report — failed, or never finished at all — returns `blocked` -without running. Raising would make Dask re-raise in the driver and abort -every task in flight, and reporting all independent failures in one run -is most of what owning the loop buys. `worker.materialize` wraps the -whole unit, so the contract holds for failure modes nobody enumerated; -the one inner guard that remains exists because "your recipe failed" and -"your recipe worked and we could not record it" deserve different words. - -**`data_version` is computed in the worker, before anything is staged.** -The dependent's argument *is* the upstream worker's return value, so the -digest has to exist at return time — when the files are still untracked -and unannexed. Deriving it from `git annex find` instead was the first -instinct and is wrong twice: nothing is annexed yet, so every output would -record `sha256([])` — one constant, silently disabling the whole chain -with green tests — and it would make the digest a function of -`annex.backend`. The annex backend is therefore not load-bearing and is -not pinned. - -**A skip returns the *recorded* digest, never a recomputed one.** On a -clone that has fetched no annex content the files are dangling symlinks, -and rehashing them would quietly report a different output. `--check` -obeys the same rule for the same reason — it reads each unchanged -upstream's manifest rather than hashing the directory, or it would report -a rebuild for a project that is entirely up to date. The honest -consequence: a hand-edited-and-committed output does not cascade in this -layer. Catching that is `lc verify`'s job. - -**Three states, and the line between them is the layer's whole shape** -(`assets.classify`). - -| state | means | what happens | -|---|---|---| -| `stale` | the artifact **contradicts** the project: the spec defines it differently than it was made, or it records deriving from bytes the project no longer holds | remade | -| `behind` | it is still exactly what the spec asks for; only the **environment** moved | reported, left alone | -| `current` | neither | nothing | - -The distinction is *contradiction* versus *circumstance*. A stale artifact -is mislabelled — what is on disk is not an instance of what the spec -declares — so keeping it would be a lie. A behind artifact is not wrong in -any way; its environment is recorded and its commit reconstructs that -environment, so remaking it buys nothing and can cost a week of -allocation. - -**`Verdict.calls_for_a_remake(refresh=)` is the one place that turns a -state into an action**, and it has three callers — the worker, `check`, -and the walk that feeds the cascade. `stale` always; `behind` only when -asked. Do not re-spell it inline; the third copy is where they start to -disagree. - -**One classification rule, and the two callers differ by one value** -(`assets.classify`). It compares `definition_version` against the -manifest, the declared input *set* against the recorded one, each recorded -`input_versions[…]` against the version it is handed, and finally -`env_version`. The set comparison is separate on purpose: -`definition_version` hashes the recipe and the decisions, neither of which -an input the spec no longer declares moves — so without it a dropped -dependency leaves the output reporting current forever. The worker hands -live digests; the read-only walk hands `None` for anything it has already -decided will run, meaning "this is going to change". That single value is -the entire difference between them — one input, conservatively chosen, not -a second body of logic — the same discipline as layer 1's -`converge(write=False)`. It is the one place in the layer where a bug is -quiet rather than loud, which is exactly why it may not have two -implementations. - -**`stale` wins over `behind` when both apply.** The artifact is going to -be remade either way, and reporting "left alone" about something the run -is about to rebuild is the one wrong answer. - -**`behind` does not propagate, and a behind upstream still feeds its -dependents.** It says the environment moved, not that the bytes are wrong, -so `TaskResult.usable` includes it and `data_version` flows on unchanged. -Propagating it would mark one old artifact's entire downstream forever and -kill the signal; the mosaic — how many environments and commits an output's -ancestry spans — is a project-level question, not a per-output flag. - -**`--refresh` widens a run by exactly one state.** It is not an escape -hatch (it asks for *more* work, not less), and it must not become a -rebuild-the-world flag: a `current` output stays current under it. There -is deliberately no flag in the other direction — nothing suppresses the -rebuild of a stale output, because deleting the directory is the user's -own file operation and is stronger consent than a flag. - -**`up_to_date` does not count `behind`, and is not true of a run that -failed.** `lc materialize --check` is a gate, and a project of curated -results would otherwise never pass it again — that is the `behind` half. -The other half is that `made` stays empty when *every* recipe fails, so -`not made and not planned` reported "nothing to do" over a list of -failures. It is `ok and not made and not planned`; those two are the -first keys of the JSON report and are what an agent branches on. - -**`lc status` reports; `--check` gates.** Status always exits 0 — a state -is not a failure — and it is the only verb that shows the commit each -output was made at, for every state and not only the interesting ones. -It reads manifests and hashes inputs; it runs nothing, commits nothing, -and does not mind a dirty tree, because the moment you most need to know -what state a project is in is when it is not clean. Two verbs answering -the same question with different exit codes is how a script comes to -depend on the wrong one, so keep the split sharp. - -**A read-only verb never tracebacks, and an entry point never -misattributes.** `lc status` and `--check` read projects that are *in a -state* — that is what they are for — so anything `_predicted` cannot read -becomes the same `None` an absent input already produces: it will be -remade, and the recipe is where that failure belongs with a real error. -(The concrete way in: `data_version`'s directory walk keeps dangling -symlinks deliberately, so an unfetched annexed file cannot drop silently -out of a digest — one that is *not* an annex link then reaches `open()`.) -The same rule at the other end: `worker.main` is what every -`[DATALAD RUNCMD]` record names, and its "no output ``" message covers -the task lookup **only**. It once wrapped the whole body, so a `KeyError` -raised anywhere inside astra's validation or resolution was reported as a -bad target — a rerun misdiagnosing itself, at the one place nobody is -watching. - -**Known violation, unfixed: a malformed `astra.yaml` tracebacks out of -`lc status` and `lc materialize --check`** as a raw -`yaml.scanner.ScannerError`. `plan._validate` is the gate that turns a -bad spec into ASTRA's own errors, but it only catches what *validation* -reports — a spec that does not parse raises inside -`validate_analysis_schema` → `astra.helpers.load_yaml` before there is -anything to validate. (`lc materialize` usually masks it by refusing a -dirty tree first, which is not a fix.) The shape of the fix is a -`yaml.YAMLError` catch in `_validate` naming the file and the line, the -same way the validator's own errors are rendered. Pre-existing and -independent of the one-output-one-file change; written down here because -it contradicts the invariant directly above it. - -**`git_sha` in a manifest is the commit the run *started* at**, not the -commit the run went on to create. It is the code that produced the output. -A test that reads `dataset.head()` after materializing and expects a match -is asserting the wrong thing. - -**One *project* uv hop, one spelling** (`project.uv_prefix(root)`), -always `--no-sync`: the driver converges the environment before it -submits a probe or a recipe, and a per-task sync would have every -concurrent worker writing the same `.venv`. - -The run record's `cmd` is the deliberate second shape, and it is not the -drift the rule guards against: it is *project-less* by construction -(`uv run --no-project --with lightcone-cli==`), so it shares no flag -with `uv_prefix` — no `--project`, no `--locked`, no `--no-sync`, -because there is no project environment involved. It builds an engine to -run, where `uv_prefix` enters an environment already built. Routing one -through the other would mean a helper with two disjoint output shapes. -What keeps *that* hop from drifting is that the worker it invokes -converges the project environment itself, so the record never has to -spell how. - -**A run syncs the environment; it does not report on it.** `uv run ---locked` asserts only that `uv.lock` still matches `pyproject.toml`, and -workers pass `--no-sync` — so a lock edited without a sync would leave -recipes importing packages the lock does not describe while every manifest -recorded the *new* lock's `env_version`. Measured: the recipe imported -`packaging 26.3` under a lock saying `24.2`, and uv accepted it silently. -`materialize()` converges right after the dirty check — before the graph -— so the state is made impossible rather than detected. (The dirty check -moved *in front* of the converge with layer 6: in containerized mode the -converge can commit an image archive, and `dataset.save` stages scoped -but commits the whole index, so on a dirty tree the user's staged edits -would be swept into the image commit. Direct mode is order-insensitive — -its sync touches only ignored paths — so both modes run one order.) -`--check` needs neither: `env_version` is the lock's bytes, so a drifted -`.venv` cannot change what it answers. - -**A run fetches its declared inputs; the read-only verbs never do.** -`materialize` batch-runs `git annex get` over the graph's in-tree -declared inputs before workers hash or execute (driver-side — the storage -invariant that nobody is ever asked to run an annex command by hand), -so a bytes-free clone materializes straight to up-to-date. A failed -fetch is a *warning*, never a refusal: independent tasks still run and -the task whose input is unreachable reports its own failure. `--check` -and `status` stay transfer-free — there an unfetched input is a -reported fact, and the report now says `lc materialize` resolves it. -Out-of-tree inputs are not fetched (no annex holds them — the recorded -weaker promise), and `test_check_mode_never_fetches` pins the read-only -half. - -**A run's concurrency is the cluster's, and there is no flag to say -otherwise.** How much of a machine a run may use — and which machine — is -one question, answered by the allocation it borrows (each node's worker -runs `task_slots_per_node` tasks at once) rather than by a `--jobs` knob. - -**A dirty tree is a refusal, and `--check` is exempt.** Every -materialization is committed with the code that produced it, so a run that -started dirty could not say what that code was. Check mode does not -refuse: reading the state of a project before deciding what to commit is -what it is for. The refusal splits by path class because the remedies are -opposite — work the researcher owns gets committed, and anything under -`results/` is lc's to write and is wreckage to discard. - -**A run leaves the tree exactly as clean as it found it.** The worker -resets the output directory *before* executing, so a failed recipe, a -crash, or a Ctrl-C would otherwise leave tracked files deleted or -half-written — and the next run's refusal would tell the user to commit -truncated, manifest-less garbage into `results/`, destroying the one -property the layer exists for. So `ok` → `dataset.save`, and `failed` or -`blocked` → `dataset.restore`. The one exception is an interrupted run: a -task that never reported may still have a recipe writing on the cluster, so -its files are left in place rather than restored underneath it, and the -dirty-tree refusal's `results/` block says to stop the allocation before -discarding them. This is what makes the refusal survivable rather than a -trap. - -**The run record names declared paths, never resolved ones.** Every -declared input under `data/` is an annex symlink, so a `Path.resolve()` -in the record's `inputs` writes `.git/annex/objects/SHA256E-…` — the -storage rather than the input, and a path nothing can `datalad get`. This -shipped once; `plan.declared_path` is lexical now, never resolved. - -**The run record is genuinely re-runnable, and every record pins its -engine.** `uv run --no-project --with '' -- python -m -lightcone.engine.worker /` — an ephemeral -environment that reconstructs the *engine*, while the worker itself -reconstructs the *project* environment from the rerun commit's own lock -(`main()` syncs before anything executes; `uv run --no-sync` against a -missing `.venv` silently creates an empty one, so a clone's rerun would -otherwise run recipes in a bare environment under a manifest recording -the lock's `env_version`). The bare recipe would reconstruct nothing lc -adds (no locked environment, no boundary, no gates, no manifest) and -would commit bytes the identity model never produced; `lc materialize` -cannot be it either, because `datalad rerun` removes the declared outputs -first and that dirties the tree materialize refuses to start from. - -The requirement (`materialize._engine_requirement`) pins a release by -version and a dev build by its source commit — hatch-vcs embeds the -commit in the version, and the repository URL comes from -`[project.urls]`, the engine's own metadata rather than a constant. So a -rerun works during development too, against the commit that ran -(verified against GitHub: uv resolves the short sha and hatch-vcs builds -the matching version from the clone). An unpushed commit fails a rerun -loudly at resolution, which beats silently finding another engine; a -dirty tree pins the last commit, and the version's `.dYYYYMMDD` marker in -the manifest is what says the bytes had drifted from it. The e2e rerun -tests monkeypatch the requirement seam to a wheel built from the working -tree (`UV_FIND_LINKS`), because a git pin can only ever build *committed* -code and the suite must execute the code under test. - -**The worker module is not an `lc` verb and not a console script.** It -makes the output unconditionally, commits nothing, leaves the tree dirty by -design. `lc --help` advertising it would hand people a footgun, and a -`[project.scripts]` entry would put it on `$PATH` through -`uv tool install`. `lightcone/_sandbox_exec.py` is the same shape for the -same reason. Keep it cheap to import — **no click, no rich** — it is on -the path of every task and every rerun. -`test_the_worker_module_imports_neither_click_nor_rich` pins the imports, -`test_help_does_not_advertise_the_worker` the absence from `--help`, and -`test_the_worker_and_the_shim_are_never_console_scripts` the absence of -a `[project.scripts]` entry. - -**The record's format is datalad's, so it is tested through datalad.** -`get_run_info` matches with a regex and returns `(None, None)` on any -mismatch, after which `rerun` says "no command; skipping" and **exits 0** — -a golden test over our own JSON would stay green through a silent break. -So the suite asserts through datalad's parser *and* runs a real -`datalad rerun`, and `datalad` is a **dev** dependency only. Nothing in lc -imports it. - -**A policy describes the project; it never prepares it.** `exec_policy` -creates nothing in the project tree — the worker resets the output -directory, so the whole of an output's lifecycle stays in the module -that owns it, and `results/` is granted only if it already exists. The -one thing it does put on disk is the per-run private `$HOME` -(`mkdtemp`), and that is why `sandbox.scope` takes a *built* policy: the -`rmtree` of that directory has one owner. (`wrap` stays pure; the -impurity lives in policy construction, once.) - -**There is one policy, `exec_policy`, and it differs between a recipe -and a probe by one keyword.** The tree is read-only apart from the -in-tree write scope: a recipe's is **the directory its output lands in** -(`write_dir=`, handed down from the worker's task), a probe's is the -project's own `results/` whole, because a probe has no analysis node. - -That is a *directory* and not the output file, by mechanism rather than -by choice: `policy._declared`/`_existing` drop paths that do not exist, -so a file scope would leave a recipe with no in-tree write at all, and -the Landlock shim masks a non-directory grant to `_FILE_ONLY_BITS`, -stripping `MAKE_REG` — a file grant could not create the file. So the -one-output-one-file change reverses the hardening pass's narrowing: -outputs declared side by side are mutually writable again. Recorded, not -papered over — what answers whether an output's bytes are its own is -`data_version`, never the sandbox. What the scope still excludes is -every *other* results tree: another universe, and another analysis's -own. - -**`cluster_for_run(cluster_id)` is the execution seam.** It borrows a -standard client and returns `submit` and `completed`. Dask chooses workers and -orders dependencies. There is no additional execution wrapper for worker selection, -source fingerprinting, or per-worker preflight. Keep the deployment's shared -project/environment prerequisites documented. Git and convergence belong to the -driver; runtime gates and sandboxing belong to tasks. - -### Recorded deviations from the spec (layers 2 and 4) - -- **`git_dirty` is not written.** Spec §3 lists it; the start-of-run - refusal makes it constant. The limitation that comes with that, stated: - the check is at start of run while manifests are written per-output much - later, so a user who edits `src/fit.py` while a long graph runs gets a - manifest whose `git_sha` no longer describes the code that ran. Since - the hardening pass the run *says* so — one `dataset.status` call after - the consume loop, warning with the edited paths (the tree started - clean and save/restore keeps `results/` clean, so any dirt appeared - mid-run) — but still records nothing in the manifest: the driver does - not rewrite files the worker owns, and per-output attribution would - need a per-save probe nothing has asked for yet. -- **The manifest carries what this layer can honestly fill.** - `schema_version`, `output_id`, `universe_id`, `recipe`, - `definition_version`, `env_version`, `data_version`, `decisions`, - `input_versions`, `git_sha`, - `git_remote`, `lc_version`, `hermeticity` — and, since layer 6, - `image` (`{tag, id, archive, arch}`, defaulting to ``None`` because - that is the true value for a host run — not back-compat machinery, - and `SCHEMA_VERSION` stays 1 pre-release), and since layer 8 - `started_at` / `finished_at` (ISO 8601 UTC, **millisecond** precision - because RO-Crate consumers parse `endTime` with at most three - fractional digits; attestation like `lc_version`, defaulted `""`, - never read by `classify`), and since the hardening pass `uv_version` - (probed once per run by the driver — `project.uv_version` — and - handed down, the HEAD discipline; the rerun entry point probes its - own; empty on failure, attestation must not fail a run). Spec §3's - remaining list — - `platform`, `worker_runtime`, `python_build`, - `dpkg_snapshot_sha256`, `sdist_built`, `env_snapshot`, `gpu_driver` — - is attestation nothing here reads; it lands with the verb that reads - it (`worker_runtime` is additionally derivable from - `hermeticity.mechanism`, so it may never land at all — the hardening - pass considered `platform` too and took only `uv_version`, the one - field the engine-closure decision named as its concrete loss). -- **`env_version` has four terms.** Layer 6 added the image term — - the system layer's identity document, hashed as the literal `null` - for a direct project so the formula stays one formula. (The spec's - separate `Containerfile.extra` term went with the concept; see the - layer-6 decisions.) -- **A recipe is handed to `bash -c`.** ASTRA recipes are command lines, - not argv, and `bash` is in the exec allowlist — so it is granted by the - same rule that grants everything else a recipe may run. - -## Key Invariants (layer 5) - -**The seam is a pure argv rewrite.** Every mechanism is a -`Backend.wrap(policy, argv) -> argv'` — a function turning a command into -*a different command that sandboxes itself*. Seatbelt is natively that -shape; Landlock is not (it is a self-restriction), which is exactly what -`lightcone/_sandbox_exec.py` exists to fix. Because both reduce to argv, -`boundary.run` never branches on platform, and **every backend is fully -testable on a host that cannot run it** — `tests/test_sandbox_wrap.py` -checks the Landlock wrap and the generated SBPL on whatever machine is -running, with no privileges. Keep `wrap` pure: no temp files, no file -descriptors, no global state. `tests/test_sandbox_wrap.py::test_wrap_is_pure` -pins it. - -**`detect()` in `sandbox/boundary.py` holds the only `sys.platform` -branch.** Everything downstream branches on `Capability.kind`, a *value*. -Adding bubblewrap or podman later is one module plus one line there. - -**Three types, deliberately distinct** (`sandbox/model.py`): `Policy` is -*what we will enforce* (mechanism-free path sets), `Capability` is *what -this host can do* (the probe's answer), `Attestation` is *what was -actually enforced* (spec §7's manifest field, derived from the flags -applied — never from what the mechanism matrix says should have -happened). Collapsing any two of them is how a sandbox starts lying. - -**`Unavailable` is a real backend, not a special case.** It satisfies the -protocol, wraps to the argv it was given, and attests `fs: open`. Saying -so is the caller's job; pretending is nobody's. There is deliberately -**no flag surface around the sandbox** — no opt-out, no require, no -debug dump. Enforcement always happens where a mechanism exists, and a -host without one gets the downgrade note, not a choice. - -**A denial is never invisible.** `denial.explain()` is a best-guess -heuristic over the child's stderr and is *allowed to return nothing* — -a command can swallow the `PermissionError` or rewrap it. That is why -`denial.trailer()` fires on **every** nonzero sandboxed exit, -unconditionally. Both fallbacks are pinned in `tests/test_sandbox_denial.py`. - -**The exec tier is per-file, and read ≠ execute.** `/usr` is readable (the -dynamic linker needs it) and never executable — `/usr/bin` holds `bash` -and `latex` alike, so a directory grant there would admit every -undeclared tool on the host and leave the layer enforcing nothing. - -**Three things are load-bearing and non-obvious**, each found by a real -failure and each now pinned by a test: - -| Grant | Without it | -|---|---| -| the realpath'd **ELF loader** in the exec set | *every* dynamically linked binary fails EACCES — bash and python included | -| **read** on the uv-managed interpreter's install root | `Failed to import encodings module` — the stdlib sits beside the binary, outside the project and outside `/usr` | -| **read** on `/dev/urandom` | `failed to get random numbers` — CPython seeds hash randomization before `main` | - -**Never grant EXECUTE on a directory that could be a system prefix.** -Landlock unions rights over ancestors, so a single EXECUTE grant on -`/usr` silently outranks the entire per-file allowlist and leaves the -layer enforcing nothing — while every test still passes, because the -allowlisted binaries are exactly the ones that were going to work. This -shipped once: the venv interpreter's *install root* was granted EXECUTE -unconditionally, which is `/usr` for any venv built on a system python -(`uv venv --python-preference only-system`, most CI images, HPC site -pythons). The rule now: the interpreter **file** always; its install -root gets EXECUTE only when it is the interpreter's *own tree* (a -uv-managed store, a framework version directory — macOS framework -builds re-exec into `Resources/Python.app`), never a `_SHARED_PREFIXES` -member; and READ goes to the root regardless, for the stdlib. -`test_a_system_interpreter_does_not_make_the_whole_prefix_executable` -pins it. - -**The env overlay belongs to the seam, not to the parent and not to each -backend.** `boundary.env_argv()` composes `policy.env` into an `env K=V …` -prefix *inside* the wrap, once, for every mechanism — never merged into -the `subprocess` environment, because everything *outside* the rewrite -must keep the real one. `uv` resolves its cache from `XDG_CACHE_HOME` and -its managed interpreters from `XDG_DATA_HOME`; overlaying those for the -`uv run` hop points it at a throwaway `mkdtemp` that `scope()` then -deletes, so every probe re-downloads the world, air-gapped hosts fail -outright, and `~/.config/uv/uv.toml` and `~/.netrc` go missing. - -Two rules, and the second was learned the hard way. **If `prefix` is -outside the boundary, nothing the boundary imposes may reach it.** And -**anything every backend must do belongs to the seam, not to the -backends** — while each applied its own overlay, `Unavailable` applied -none, so an unenforced run silently got a different environment as well -as no enforcement. A mechanism added later cannot forget what it never -had to remember. - -**The shim stays alone.** `lightcone/_sandbox_exec.py` imports nothing but -the stdlib and nothing from lightcone — `lightcone` is a namespace package -with no `__init__`, so `python -m lightcone._sandbox_exec` executes that -file and nothing else. It runs on every sandboxed exec; an engine import -there would put click, rich, and the astra stack on that path. Two tests -pin it. Its setup failures exit the reserved code **97**, distinguishable -from anything a command could return, and it never falls through to -running the command unsandboxed. - -**Layer boundaries showed up as parameters we did *not* add.** -`writable_project`, output-dir write scope, and scratch dirs belong to -later layers and are absent, not stubbed. (Layer 6 then landed the -`podman`/`docker` attestation values and the containerized policy shape -— as one keyword on `exec_policy` and one backend, not as the stubs this -rule kept out.) - -**The macOS profile is vendored, not authored.** -`sandbox/profiles/{base,network,platform-defaults}.sbpl` come from the codex CLI -(itself Chrome-derived), with a provenance header naming the upstream -commit and a single `LIGHTCONE DELTA`. The macOS read baseline is not -derivable from first principles — it is a list of things that break, -found one production failure at a time: `/dev/dtracehelper`, the -`/dev/fd` and pty regexes, firmlink-parent traversal under -`/System/Volumes/Data`, the `opendirectoryd.libinfo` lookup without which -`getpwuid()` raises `KeyError`, `/opt/homebrew/lib`. Keep them -near-verbatim so `diff` against upstream stays the re-sync tool; put our -own rules in the generator, not in the vendored text. The one delta — -upstream's blanket `(allow process-exec)`, which exists because codex -does not restrict exec — is pinned by a test so a re-sync cannot silently -restore it. - -**Not controlling the network takes more than allowing sockets.** On -macOS, name resolution and TLS trust go through *mach services* -(`SystemConfiguration.DNSConfiguration`, `configd`, `SecurityServer`, -`trustd.agent`, `ocspd`), which the base's `(deny default)` blocks — -`(allow network*)` opens the socket families and nothing else. That is -why `network.sbpl` is emitted unconditionally: without it the -attestation reads `network: allowed` on a host where every lookup fails, -which is worse than either honest answer. Linux needs no equivalent -because Landlock gates the filesystem only. - -**One resolution rule for the utility allowlist**, `policy.utility()`. -Anything that hardcodes a tool's path instead is a second answer to the -same question, and the two diverge the moment a host keeps its copy -somewhere else. `env` is the load-bearing case: the seam execs it to -apply the overlay, so a stale `/usr/bin/env` is not a missing -convenience but a denial on the first exec of *every* run — `/usr` is -readable and never executable. Pinned by -`test_the_env_the_seam_execs_is_one_the_policy_granted`, which points -the search path at a copy, because a test against the real path passes -on this host and ships the bug. - -**A derived read root may never be `$HOME` or above** (`_stdlib_root`). -The interpreter's install root is granted READ for the stdlib beside it, -and for an interpreter installed straight into `~/bin` that root *is* -the home directory — silently undoing the private-`$HOME` design, which -is the whole point of the environment overlay. Reading `pyvenv.cfg`'s -`base-prefix` instead does not help: it reports the same directory. The -guard is the fix; failing loudly on a layout nobody uses beats voiding -the guarantee for the people who do. - -**The two mechanisms disagree about the write root itself.** Landlock -follows POSIX — unlinking a directory needs write on its *parent* — so a -recipe granted write on its output directory cannot remove that -directory. Seatbelt's `(allow file-write* (subpath …))` covers the -directory node, and permits it. Found by CI, which is the point of -running one suite on both. Since the hardening pass narrowed a recipe's -write scope to its own output directory, the asymmetry sits exactly on -the node a ported recipe likes to `rm -rf` as a prelude (measured on -real Landlock: `rm -rf "$OUT" && mkdir "$OUT"` is EACCES on Linux and -succeeds on macOS) — so such a recipe is green on a laptop and red on -the Linux venue. Nothing in *lc* depends on either answer: the worker -resets the directory before every execution, so the prelude is -redundant and the fix is deleting it; likewise a temp-then-rename -through a `results/` sibling now fails on both platforms — scratch -belongs in `$TMPDIR` (the private HOME), which is what the write-denial -remedy says. The asymmetry cannot be closed without granting write on -the *parent*, which is precisely the sibling-write hole the narrowing -exists to close; documented rather than papered over, and a test that -asserts one mechanism's answer will go red on the other. - -**SBPL is last-match-wins; Landlock unions.** This asymmetry decides -where a rule can live, and it cuts both ways. A later `(deny …)` can take -back an earlier allow, which is how `_read_only_guard` takes back write -on the read roots the vendored defaults would otherwise hand out; -Landlock has no equivalent — a narrower rule only ever *adds* — so on -Linux the policy simply never names them. - -But the same asymmetry means SBPL does **not** give nesting for free. -Landlock unions, so a writable output directory inside a readable project -tree just works. In SBPL the guard's `(deny file-write* PROJECT)` would -revoke it — which is why `generate_profile` restates the **write tier -last**, after the guard. Get that order wrong and layer 4 materializes on -Linux and refuses on macOS, with the golden test still green because it -only checks that the guard is present. Verified empirically, not assumed. - -## Key Invariants (layer 6) - -**Mode is derived, never configured.** A `[tool.lightcone.image]` table -in `pyproject.toml` *is* the escalation to containerized mode; deleting -it is the way back. `project.mode()` reads presence only; what the table -*means* is `engine/image.py`'s question, so an invalid declaration -refuses where it is consumed, naming the key at fault. - -**The user never sees a Containerfile.** The whole declaration surface is -Modal-shaped TOML — `base` (digest-pinned or the default constant), -`apt-install`, `run-commands`, `env` — a closed set, because every key is -hashed. `pip_install` deliberately has no equivalent: the Python -environment is the lock's business, never the image's, and `run-commands` -is the bounded escape (this deletes spec §2's `Containerfile.extra`). -The render exists only inside a transient build context; the image's -`LABEL io.lightcone.image` carries the identity document, so the archive -stays self-describing without one. - -**The engine never enters the image.** The container is the *recipe's* -execution world: driver, git, annex, dask and classification all stay the -host's `lc`, and exactly two things ever run in-image — the environment -converge (`container.sync`, project `:rw`, host uv cache mounted, into -`.lightcone/venv`) and each recipe/probe exec (mount table only — the -converge differs by its writable mounts, not by network: the network is -uncontrolled on every mechanism). This deviates from spec v6.1's full-stack rule, -recorded: v6.1's reason was the host-sync deadlock, which the -in-container sync solves, and the spec's own Perlmutter row ("recipe -wrap, step 3 only — never the dask worker") is this exact shape. What it -buys: no delegation machinery, no git through a `--userns=keep-id` bind -mount, no engine version in the tag, and one engine per run by -construction. The host `.venv` is inert in containerized mode -(`current_project` does not require it; `lc init` converges no -environment — a host sync of a containerized lock is the deadlock in -miniature); `.lightcone/` is gitignored machine state. - -**The image is the system layer only.** Base + apt (only when -`apt-install` is nonempty — the engine needs nothing from apt, so spec -§2's rule is restored) + the pinned uv + the pinned interpreter into -`/opt/python`. No git, no engine, and **no project file ever enters the -build context** — the context is a scratch directory holding one rendered -Containerfile, which is what makes "code edits never trigger a build" -structural. `bash` is a base-contract check (recipes are `bash -c`), and -each contract violation is a reserved exit code mapped to a refusal -naming the base (43 musl, 44 no bash, 45 no apt), never a raw build log. -The readability `chmod` rides inside the layers that write `/opt` — a -layer of its own would copy-on-write the whole interpreter tree into -every archive, doubling it. - -**The dataset is the image store; runtime stores are caches.** `lc build` -saves the image as a `docker-archive` at -`.datalad/environments//image` — the `datalad containers-add` -layout — annexed and committed, so the exact bytes travel through -`git annex get` with no registry and no credentials. This supersedes the -spec's `dpkg_snapshot_sha256` residue: apt is name-pinned, so a rebuild -months later yields different bytes under the same tag, and the archive -keeps the bytes themselves. The dot-path is the trap: -**git-annex routes dotfiles to git whatever `annex.largefiles` says**, -so `dataset.save` opts in unconditionally (`annex.dotfiles=true`, -per-add like `annex.thin`) — without it the archive, or a `.cache.h5` a -recipe writes into its output directory, lands as a full blob in git, -silently, with every test green. The attributes alone decide from -there (dot-named manifests keep their own exemption); a user's plain -`git add` keeps git-annex's stock behavior. `test_dataset.py` pins both -directions. And because `.gitattributes` is user-authored and lc only -appends, the build **probes the routing with `git check-attr` before -saving** — an archive the attributes would hand to git is a refusal -naming the line, the same probe-don't-assume rule as the `results/` -ignore check. The archive -is the arch it was built for (recorded in the manifest); multi-arch is a -layer-7 item, as is apptainer/singularity — which is *why* the format is -`docker-archive`: all four runtimes consume it, and HPC hosts that -cannot build obtain images through the repository. - -**`runtime_for_run(root, *, build)` is one function with two -strictnesses**, and the split -is load-bearing: `lc build` and the materialize preflight may **build + -save + commit** (announced by the CLI, which owns the console — the -engine never prints); the probe and the worker entry point only ever -**find** — a missing archive refuses naming the exact `lc build` -(`lc run` never builds; the worker never commits), unfetched content is -**fetched by lc itself** (`git annex get`, driver-side — the storage -invariant is that nobody is ever asked to run an annex command by hand, -so the refusal is reserved for a fetch with no reachable copy), and an -unloaded image is a silent ` load`. Execution pins the **id** — sha256 of the -archive's config blob, readable with no runtime, the same computation -datalad's docker adapter makes — never a tag, so a retagged store image -cannot substitute. **A dropped archive never substitutes**: a rebuild is -a new archive commit under a new id; old manifests keep naming what ran. - -**Builds and the archive commit happen only on a clean tree, and only -after the graph.** The dirty check runs before `runtime_for_run` in -materialize, and `lc build` refuses dirt itself: `dataset.save` stages -scoped but commits the whole index, so a build on a dirty tree would -sweep the user's staged edits into the image commit — and the tag -derives from `pyproject.toml`, so the declaration must be committed -before the image it defines. The graph (spec validation, the lock scan) -resolves before the image too, in materialize and in the worker entry -point alike: a refusal over a typo must not cost a minutes-long build or -leave an archive commit behind a failed run, and a failing sync must not -bury "no output `x`". - -**Two identities, one document.** The canonical identity document -(declaration resolved + the uv digest; the interpreter pin deliberately -absent — its raw bytes are already an `env_version` frame) feeds -`env_version` as its fourth frame, `null` for direct projects — so -declaring an image, or an engine release that bumps the default-base or -uv constants, puts containerized outputs `behind`, honestly, while -direct projects never move on an engine release. The **tag** hashes the -rendered Containerfile *and* the document, so a render-only generator -change rebuilds the image without staling anything — accepted residue, -attested by the manifest's image id and the archive's bytes. Read from -`pyproject.toml` only, never `identity._uv_config()` — uv's -"`uv.toml` replaces `[tool.uv]`" rule must not reach this table. - -**The mount table is the mechanism** (`sandbox/oci.py`). The -containerized `exec_policy` shape is the same policy minus the host: no -OS read baseline, no stdlib root, no exec set — the image *is* those, -and everything in it was declared — leaving exactly the paths that -become mounts, project `:ro`, the write scope `:rw` (a recipe's own -output directory; a probe's `results/`), declared inputs `:ro`, -the private HOME `:rw`, `--tmpfs /tmp`, over a **`--read-only` rootfs**: -without that flag a write outside the declared set *succeeds* into the -container's ephemeral layer and vanishes while the run attests -`fs: declared` — the silent-loss path, closed by making it a denial. -Mounts are **resolved source, declared destination** — the containerized -policy is the one shape that keeps its paths unresolved, because they -are addresses the recipe uses (a symlinked `/data` input resolved on -both sides would leave the container with no `/data` at all). -`--security-opt label=disable`, because SELinux hosts otherwise refuse -every bind read and `:z` would relabel the user's own files. No -in-container Landlock, no seccomp probe, no shim-in-image: the engine -container never gets the tree `:rw`, so mounts alone express the whole -policy, and the attestation (`mechanism: podman|docker`, -`fs: declared`, `network: allowed`) is derived flag-for-flag from the -argv — no flag touches the network, the same non-control every -mechanism attests. One `OCIBackend`, data-parameterized: podman and docker differ -in spellings (`--userns=keep-id`+`--pull=never` vs `--user uid:gid`), -not shape. -Runtime is **host capability** — detected podman → docker, the -`detect()` discipline on a second axis, with `container.backend()` -holding the only mode branch — and never part of any identity. The -containerized `tmp_home` lives under the project's `.lightcone/` rather -than the system temp dir: it is a mount source, and macOS's podman -machine shares the project tree while `/var/folders` arrives empty. - -**The seam learned one asymmetry, `contains_prefix`.** A host mechanism -keeps the `uv run` hop *outside* the wrap (trusted host plumbing); a -backend that is itself a world takes it *inside* — there is no trusted -host plumbing inside a container — and applies the env overlay natively -as `--env` flags, never through `boundary.env_argv`'s host-resolved -`env` binary (NixOS resolves it to a path no Debian image has). Declared -on every backend, so a new mechanism must answer the question. The -container environment is an allowlist: the overlay plus `LC_SANDBOX`, -never the ambient environment. - -**The record stays runtime-neutral, and the worker is its executor.** -`cmd` is still the engine-pinned worker — verified against -datalad-container's source before deciding: `containers-run` expands its -`cmdexec` template at *record* time and `datalad rerun` executes the -literal string (the extension has no rerun hook), so a bare-recipe `cmd` -would rerun on the host, unconfined, with no manifest. Instead the -record is declarative — the task id, the commit's own spec, the -committed archive — and the worker re-resolves container *and runtime* -on the rerun host, keeping the sandbox, the gates and the manifest. The -containerized record adds `extra_inputs: []`, which stock -`datalad rerun` fetches through the annex before executing — the smoke -suite proves the whole claim on a bytes-free clone. `lc build` also -writes `datalad.containers..{image,cmdexec}` so ad-hoc -`datalad containers-run` works for humans, explicitly outside lc's -guarantees. - -**The runtime is resolved once per run and handed down** — the HEAD -discipline, a frozen `container.Runtime` through `worker.materialize` — -because resolving per task could answer differently mid-run. The rerun -entry point resolves its own, as it does HEAD: it *is* the driver of its -one-task run. `lc status` gained the header lines (mode, image -state, sandbox — and, since the hardening pass, crate) — repository -facts only, no runtime and no network -required, which is where the denial note and the runtime-missing -refusal point. - -## Key Invariants (layer 7) - -**Compute is explicitly allocated and borrowed (2026-09).** This supersedes -the ambient venue ladder. `lc compute resources/launch/status/down` manages local -and Slurm allocations through `engine.compute.Provider`. `run` and `materialize` -require a cluster name or full ID as their first positional argument; neither creates compute. -`materialize --check` remains cluster-free. Catalog offers expose resource shapes -and name their provider; each provider reaches one native authority. Native jobs and validated local OS -identities are the allocation authority; standard Dask supplies execution state. -No Lightcone server, lifecycle database, custom Dask worker, or implicit allocation. - -**Names are native labels, not a registry.** `launch --name analysis` chooses a name; -otherwise the local shortcut uses `local`, and explicit resource requests generate -`lc-` plus 12 random hexadecimal characters. Plain stdout -contains only the name for shell capture; JSON retains the full immutable ID too. -Resolve names through fresh discovery and refuse missing, ambiguous, or incomplete -observations. Check existing names before submission, but do not claim atomic global -reservation across native backends. A name can be reused after termination; use the -full ID to address an exact incarnation or bypass unrelated discovery failures. -Slurm uses `JobName=lc-` and -`Comment=lightcone:kind=dask:token=<32hex>`. Verify the owner and both native -fields before attachment or cancellation. Missing live comments make discovery -incomplete; missing historical comments leave identity unknown, so a user's own -job named `lc-…` blocks name checks until it is renamed or ends — loudly, -naming the job, never by adopting it. Historical -comment retention requires Slurm's `AccountingStoreFlags` to include `job_comment`. -Resolve the Slurm command user's UID through `id -u` on the same command runner, -and use it for every native ownership check and filter. - -**Compute formats carry no version (2026-10).** Cluster IDs, native labels, -allocation records, the catalog and compute JSON output are unversioned: -allocations are ephemeral, so a format change strands only allocations that -end on their own, and new ones are made under the new format. Manifests are -different — committed, they keep `schema_version`. - -**Local compute needs no setup.** The built-in local offer provides detected usable -logical CPUs and RAM, one node, fast startup, no walltime and a 30-minute idle -timeout. Loading the catalog writes no catalog and starts no cluster. -`lc compute launch` without CPU/memory flags selects only local offers and defaults -the name to `local`. `--wait` returns when the accepted allocation is ready; timeout -or startup failure retains its ID without resubmitting or terminating it. -Configured remote offers precede the built-in local offer in selection order. -A catalog's own local offers replace it, which is the one way to change the -local CPU/RAM budget or time limits: there is no separate local settings block. -`allow_local: false`, the catalog's only local key, blocks local launch and -execution while preserving inspection and termination. Recognized NERSC login nodes disable local compute automatically; -other sites can disable it in their catalogs. Native permissions remain the -enforcement boundary. -The built-in local offer takes its GPUs from the `CUDA_VISIBLE_DEVICES` mask on -Linux, counted as CUDA reads it (up to the first entry that is neither an index nor -a device UUID, so `-1` is none), labelled generic `GPU`; without a mask it is -CPU-only. Remote GPU offers still require an explicit catalog, and an explicit -local GPU offer plans only when the mask exposes at least its count. No GPU -auto-discovery: the mask is the allocation, never probed against hardware, so local -GPU capacity and model labels are configured, not hardware-verified; allocations do -not reserve devices exclusively against other host programs or allocations. The -local shortcut takes an offer whole, GPUs included; `--gpus 0` takes it without -them, since local GPUs are never reserved. -Missing explicit paths and invalid files are errors. Execution still requires an -explicitly launched cluster's name or ID. - -**A Slurm offer needs no launch settings (2026-09).** Every launch key in an -offer's `config` defaults, and the defaults assume a home directory shared by login and compute -nodes, which is also what SkyPilot's Slurm backend assumes. Workers run -`sys.executable`, the driver's own installation, so client, scheduler and workers -match exactly with no resolution and no network on compute nodes. Launching the -bootstrap through `uv run --with` at job time was considered and rejected: it -re-resolves the Dask closure away from the driver's, and it needs package-index -access from compute nodes. SkyPilot installs its runtime per node only because its -client is off-cluster; lc submits from the login node, where its installation -already is. The catalog's top-level `connection_root` defaults to `~/.lightcone/compute` -(`runtime.DEFAULT_CONNECTION_ROOT`, shared with local). An unset `scratch_root` -is chosen by the bootstrap on each node (`tempfile.gettempdir()`), never frozen -from the driver's temporary directory. - -**Slurm supervises workers through stock Dask Nannies (2026-09).** Each rank -runs a Nanny and its separate worker process; rank zero also hosts the scheduler. -Nanny defaults `OMP_NUM_THREADS`, `MKL_NUM_THREADS`, and `OPENBLAS_NUM_THREADS` -to `1`, preserving explicit launch environment values, and sandbox policy passes -the effective values into recipe containers. The step sets -`--kill-on-bad-exit=0 --wait=0` so a rank exit alone does not terminate the others. -This is worker-process recovery, not complete recipe isolation: site OOM policy -may still kill a step or job; dead schedulers and Nannies are not restarted; -Dask retries are not fenced from surviving recipe subprocesses. Keep -`memory_limit=0` because Dask does not account for subprocess RSS. New commands -still require all expected workers. Serial per-output Git/annex commits remain -the driver's responsibility; changing that persistence/provenance model is deferred. - -**Compute uses one shared Pydantic model family.** `Catalog` loads directly into -the `Offer`, `Resources`, `TimeLimits`, and `Startup` objects used by -providers; do not introduce parallel configuration classes. Memory units are explicit: -`Resources.memory_gib` / `memory_bytes`, `Resources.from_bytes(...)`, and -`Request.memory_bytes`. Duration strings expose derived seconds through `TimeLimits`. -Use validated `replace` -for updates and the explicit `as_dict` allowlists for public output. Preserve -duplicate-key rejection in YAML; providers validate their own offers' `config`. - -**Allocation syntax follows SkyPilot without depending on SkyPilot.** Compute -CPU/memory requests support exact quantities or `+` minimums. Compute memory -uses binary units: bare `32`, `32GB`, and `32GiB` agree. Catalog resources use -one `accelerators: NAME[:COUNT]` or a one-entry mapping; CLI `--gpus A100:4`, -`A100`, or generic `GPU:4` selects an exact positive whole count, while `0` -means CPU only. Type matching is case-insensitive; no GPU `+`, fractions, or -global model alias registry. Local accelerator labels are trusted configuration. -Named Slurm offers must map their public label to the site's GRES type through -`config.gpu_type`; generic `GPU` offers may omit that setting. Preserve native -evidence in observations. - -**Configured compute roots may be filesystem aliases.** Resolve connection and -scratch roots before appending managed provider, submission, or attempt paths. -The catalog resolves `connection_root` once, at load, so an unusable root is a -catalog error naming the field; providers append to it and never resolve it again. -Keep symlink rejection within those managed paths and enforce private directory -and credential permissions. Do not resolve Python executables: virtualenv paths -must retain their environment identity. Never change existing ancestor permissions. - -**Slurm chooses the partition unless the catalog supplies one.** Submit a positive -native `--time` request without choosing a default partition or inspecting site -configuration during planning. NERSC routes requests by QoS and constraint. -Walltime follows Slurm's native overrun and termination-grace policy; Lightcone -does not independently guarantee a finite termination deadline for Slurm jobs. - -**Execution borrows a client and leaves the allocation alive.** Validate native -identity and scheduler readiness. The driver keeps git and convergence. Use unique -invocation task keys. Interrupted unreported outputs remain in place because a -client disconnect does not prove remote subprocess termination. Comprehensive -cancellation/fencing and simultaneous writers are deferred by explicit user decision. -Local containerized processes can outlive process-group shutdown; do not claim -that `down` or walltime proves an external runtime's containers have stopped. -Read-only project validation precedes cluster connection, and a run with no -tasks never connects: it only converges the crate. Populate the declared -input-hash memo on the driver before serializing it to independent worker tasks. -Any driver failure while tasks are outstanding (a failed commit included, not -only a cluster error) carries `compute.UNSTOPPED`, the one wording for "the -allocation was not stopped and unreported tasks may still be running". - -**Recipe resources use standard Dask admission.** Preserve ASTRA `recipe.resources` -in `plan.Task` as raw mappings so `status` and `--check` remain independent of -executor support. Parse `TaskResources` at execution admission: whole CPUs, memory -bytes, and whole GPU counts. Reuse the read-only classification walk before -admission: known current/behind outputs become values without Dask submission. -Validate tasks that may execute, including dependents of potentially rebuilt -outputs; workers recheck actual upstream digests. Normalize worker budgets once -with `worker_capacities`, then pass reservations explicitly to submission. Recipe -`time_limit` is unsupported and must fail explicitly; allocation walltime remains supported. -Workers advertise CPU/MEMORY/GPU; tasks reserve their declarations. Omitted RAM -adds no memory reservation; CPU requests and task slots govern concurrency. -Probes reserve all whole-worker budgets. -GPU recipes reserve the worker's full GPU budget, one GPU recipe at a time, and -inherit its whole allocation mask. Recipe `gpus` is a minimum capacity requirement, -not a visibility limit; it defaults to zero and does not select a model. Recipe -memory retains ASTRA units (`8Gi` binary, `8GB` decimal, no bare quantities), -independently of compute's SkyPilot units. -Thread slots remain a separate concurrency cap. Reservations are cooperative, not -per-command OS CPU/RAM limits or BLAS thread counts. Local and Slurm Nanny defaults -keep OMP/MKL/OPENBLAS threads at one unless the launch environment overrides them; -sandbox policy forwards the effective values into containers. Unsupported -disk/model requests and fractional CPU/GPU counts fail explicitly. Exact bytes are shared in `units.py`; -allocation durations are parsed in `compute.model`, with error conversion only -at the CLI request boundary. - -**GPU visibility comes from the allocation, not device discovery.** No CUDA probe, -UUID inventory, MIG detection, model verification, or custom Dask worker. Local -launch freezes the externally supplied nonempty CUDA mask and optional device -order. Slurm validates native GPU counts, preserves its mask, and sets -`CUDA_DEVICE_ORDER=PCI_BUS_ID`. `exec_policy(use_gpus=True)` inherits that whole -mask; CPU commands get an empty one. Never mutate the reusable worker's environment. -Direct GPU policies grant native NVIDIA character nodes; OS permissions and cgroups -remain authoritative. The host must initialize NVIDIA character devices including -UVM before launch; lc neither loads drivers nor creates nodes. Standalone GPU -reruns need an explicit CUDA mask in their own environment. Container GPU execution -supports podman-hpc `--gpu` only. Explicit GPU recipes on ordinary Docker/Podman -fail before image preparation; probes use CPU policy with a diagnostic note while -retaining their whole-worker reservation. CPU containers remain supported on all -runtimes and set `NVIDIA_VISIBLE_DEVICES=void`. Physical GPU execution remains -unvalidated; tests check real subprocess masks and native argv without GPU hardware. - -**One catalog selector, `LC_COMPUTE_CONFIG` (2026-09).** `lc compute --config` -was removed: `run` and `materialize` resolve clusters through the catalog too, -and a per-invocation override on one command group launched allocations those -verbs could never find. An environment variable reaches every verb. - -**Retire ended local allocations; never delete their record.** Once `down` or -discovery verifies the owner process is gone, `_retire` removes the TLS -material, scheduler files and scratch, and writes `ended.json`, which discovery -skips unread. `identity.json` stays, so a full ID still answers `ended` with -its startup error. Deleting the directory was rejected: a missing record cannot -be told from a changed `connection_root`, and treating it as ended would let -`down` return while a live owner runs to walltime. - -**Native queries stay sparse.** Slurm discovery is one `squeue` plus one -single-job `scontrol` per managed job; `status --wait` backs off from one to -30 seconds. A live job whose owner, name and token verify is cancelled in any -Slurm state; only a job absent from `squeue` must prove from accounting that it -ended. Every scheduler lc launches runs under `runtime.SCHEDULER_CONFIG`, whose -zero `events-cleanup-delay` drops a departed client's forwarded recipe output -instead of holding it for Dask's default hour. - -**Block local compute on recognized NERSC login nodes (2026-09).** A nonempty -`NERSC_HOST` plus a short hostname matching `login[0-9]+` disables local launch -and execution, including explicit local offers and `allow_local: true`. -Interactive compute nodes remain eligible; inherited `SLURM_JOB_ID` never exempts -a login node. Apply this policy at runtime without writing a configuration file. -Keep inspection, termination, and Slurm execution available. Both execution -commands submit ordinary tasks to the Dask scheduler without worker restrictions; -existing task runtime gates and sandbox checks remain. Do not add a per-worker -validation framework around ordinary Dask task submission. Shared project storage -and compatible worker installations are deployment prerequisites. - -**Remote output preserves bytes.** Standard Dask events carry bounded stdout/stderr -chunks. `run` preserves the two streams, including binary stdout and CRLF. -Materialization forwards recipe diagnostics to stderr, reserving stdout for its -report. Detached workers must not swallow a failed recipe's real error. - -**podman-hpc is a spelling, not a shape.** It rides the existing -`OCIBackend` (standard podman flags, `--userns=keep-id`, -`--pull=never`), joins the podman family in `uid_flags`, `_loaded` -(`image exists`) and `_build` (`save --format docker-archive`) — the -set stated once as `container._PODMAN_FAMILY` and asked positively at -every site, so a future runtime falls *outside* it by default instead -of inheriting podman behavior through a `!= "docker"` back door — and -adds -exactly one step: `podman-hpc migrate ` in `runtime_for_run`, -**outside the load branch** — after a fresh `lc build` the image is -already in the store and the load never runs, but compute nodes only see -migrated images. Migrate runs driver-side wherever `runtime_for_run` -runs (login node at `lc build`, materialize preflight, the rerun entry -point); one migrate serves all nodes. Detection order is -**podman-hpc → podman → docker**: a site installs the wrapper precisely -because plain podman's node-local overlay store is invisible to compute -nodes, so where both are on PATH the bare sibling is the broken one. -Presence-only detection (no daemon, no machine); the no-runtime refusal -still names only podman/docker — podman-hpc is site-installed, never a -user remedy. It is build-capable (it wraps `podman build`; NERSC login -nodes are where a matching-arch archive comes from), so all three -runtimes are both build- and run-capable and a capability predicate -would be dead code — it lands with the first run-only runtime. - -**The architecture gate refuses before the load.** A wrong-arch -`load` *succeeds* and then dies as `exec format error` deep inside a -recipe — so `_require_arch` compares the archive's recorded arch against -`platform.machine()` (via the closed `_OCI_ARCH` map) before anything is -loaded, naming both arches and the fix: build on a matching host (on -NERSC, a login node), commit, push, pull. Ignorance is not a mismatch — -an archive that does not say, or an unmapped host machine, passes. -Recorded residue: an arm64 mac that *could* emulate an amd64 archive is -refused too (emulation is unattested ten-times-slower execution), and -full multi-arch (arch-suffixed archive paths, `--platform` cross-builds) -remains open, with the venue that makes it real. - -**Design headroom for a daemonless runtime (apptainer/singularity), -deferred but load-bearing**: `Runtime` stays facts (root/mode/name/ -tag/id/arch), never mechanism — the committed→fetched→loaded ladder -stays name-branched inside `runtime_for_run`, and a store-less runtime's -analogue is a one-time archive→SIF conversion into gitignored -`.lightcone/` cache keyed by the same runtime-independent config-blob -id (the reason `docker-archive` stays the store format). -`container.backend()` stays the single construction point; a future -world-backend is one dataclass in `sandbox/` plus one branch there. -Since the hardening pass every mechanism attests `network: allowed` -with no denial flag emitted, so a store-less runtime has nothing to -imitate there. `boundary.run`'s exit-125 note is a -podman/docker-family fact, not a `contains_prefix` fact — it becomes -mechanism-keyed when a non-OCI backend lands. - -**Pending the one-time Perlmutter deployment test** (run `tests/ -test_container_smoke.py` on a login node, then one materialize through -`sbatch`; record findings here): detached salloc session lifetime and `--cpus-per-task` -behavior inside salloc/sbatch steps; `nidXXXXXX` resolution from peer -nodes (else `launch.interface: hsn0`); whether `--cpus-per-task=256` -and the task's CPU affinity agree on a CPU node (128 cores, 256 -hyperthreads); cold-Lustre `distributed` import vs the -120 s worker wait; `podman-hpc migrate` accepting a bare image id and -re-running cheaply; podman-hpc `--module` -site-injected mounts vs the honesty of `fs: declared` (the one item -that could add a flag); whether Landlock is in the SLES boot LSM list -(either answer is handled — a host without it attests `fs: open` with -the downgrade note); `git annex add`/`get` timings on `$SCRATCH` and -`$CFS` (the recorded Lustre residue). - -## Key Invariants (layer 8) - -**The project is the crate, and materialize maintains it — there is no -export verb.** `ro-crate-metadata.json` sits at the project root as a -derived artifact, converged by `lc materialize` the way `uv.lock` is: -`engine/crate.py` renders it from repository state, the hook -(`materialize._converge_crate`) string-compares it against the tree, and -only a difference is written and committed — alone, in its own trailing -commit, which `datalad rerun` skips harmlessly as "no command". Deposit -is `git archive` / `datalad export-archive` on a repository that is -already a crate; nothing is copied and there is no bundle directory. -The rerun entry point deliberately does not regenerate it (it is one -task's executor, not the driver), so the crate lags until the next -materialize — and since the hardening pass `lc status` says so: its -`crate:` line compares the document's own `datePublished` against the -newest manifest `finished_at` the status walk already read — a content -check, no git and no rocrate import (the recorded constraint that the -crate stays the one materialize-only dependency on status's path). The -`datePublished` pin is therefore load-bearing twice: it keeps the clock -out of the render *and* it is what makes the lag detectable. The line's -claim is scoped to what the proxy can see and worded to it ("up to date -**with the outputs**" / "behind **the outputs**"): a crate-affecting -edit that moves no manifest — the lock's bytes in a `File` entity, a -spec edit — is invisible here, and fine, because the next materialize -converges those anyway; a rerun's lag is the case with no other -surface, and the proxy is exact for it in both directions (a dropped -output regresses the newest stamp just as a rerun advances it). -`license_of` and `CRATE_FILENAME` moved to `project.py` so -status can ask about publication intent without the renderer's stack. - -**Publication intent is derived, never configured.** A -`[project].license` in `pyproject.toml` turns crate maintenance on — -RO-Crate *requires* a license, materialize must not refuse to run -science over a missing key, and inventing CC-BY-4.0 (the pre-rebuild -default) asserts terms over someone's data. Absent ⇒ no crate and one -report line; removed later ⇒ the file is left (it is in committed -history either way) and the line says it is no longer maintained. The -same shape as `[tool.lightcone.image]` deriving containerized mode. - -**The document is a pure function of repository state, and that is what -makes convergence sound.** The clock never enters it: `datePublished` is -the newest manifest `finished_at` (the spec file's own last-commit date -for a never-materialized project) and **must override rocrate's -default**, which stamps construction time — the old exporter carried a -`datePublished` its code never set. Entities are built in sorted order -and serialized with `sort_keys`; -`test_rendering_twice_at_the_same_state_is_byte_identical` pins it. The -render is also injected-pure: `dataset.last_writer` comes in as a -callable and `dsid` as a value, so `tests/test_crate.py` runs with no -git at all. - -**Run identity comes free from `git_sha`.** The driver reads HEAD once -per run and hands it down, so every output of one materialize carries -the same sha — grouping manifests by it *is* grouping by run, and that -is what makes the Provenance profile expressible with no new manifest -field: one `OrganizeAction` + workflow-level `CreateAction` per run, a -`ControlAction` per output execution, a `HowToStep` per output id -(deduped across universes: a step is spec structure, an action is one -execution — exactly how the multiverse maps on). - -**The crate's `Person` is the author of the output's *saving* commit, -via `last_writer` — never the manifest's `git_sha`**, which is the -commit the run *started* at and can be someone else's. No `--author` -flag, no env var, no config probe: `require_committer` already -guarantees the commit has an author. - -**The manifest stays canonical; the crate does not transliterate it.** -`env_version`, `definition_version` and `hermeticity` get no schema.org -spelling — the manifest itself is in the crate as a `File`, `subjectOf` -its output's `Dataset`, so nothing is lost and no vocabulary is -invented. What does get real vocabulary comes from the workflow-run -`@context` (`https://w3id.org/ro/terms/workflow-run`), without which -`containerImage`, `sha256` and `ContainerImage` are undefined terms -JSON-LD drops on expansion — the pre-rebuild exporter's silent failure. -The committed archive is one entity, `["File", "ContainerImage"]`, -identity (`sha256` = config-blob id) and payload together. - -**A published `sha256` is always a raw digest an outsider can verify**, and -since the one-output-one-file change the manifest agrees with it: an -output is a file, `data_version` drops the `file:` frame, and the -manifest's digest, the crate's `sha256` and `sha256sum` are one number. -Only directory-valued declared inputs stay framed (`dir:`), which is -enough to keep the two from colliding and is the one entry in a manifest -no standard tool can check. Historically — -since the hardening pass, never lc's framed hash. Every file in an -output directory is a `File` under its dataset's `hasPart`, with -`sha256` and `contentSize` parsed from its SHA256E annex key -(`dataset.annex_keys`, one `git annex find --include=*` — the -`--include=*` is load-bearing, bare `find` lists only *present* files -and the crate must answer bytes-free; the key map is injected into -`render` like the writer, so the builder stays git-free). A non-SHA-256 -backend key yields size and no digest, never a wrong one; git-carried -files hash their own working-tree bytes. Out-of-tree declared inputs -publish *no* digest: their recorded `input_versions` value is the framed -hash, which shipped once under the `sha256` term as though `sha256sum` -could check it — the manifests keep that story. The dataset's `version` -stays lc's framed directory digest, deliberately distinct from the -per-file claims. The move re-scoped what an in-tree input's `sha256` -*means*, and the old conflict rule went with it deliberately: a `File` -entity's checksum now describes the deposit — the bytes a `git archive` -carries — never what any particular run consumed, so two manifests -recording different digests for a shared input (a half-rebuilt project) -no longer suppress it; which bytes a *run* consumed is its own -manifest's `input_versions`, in the crate as that manifest `File`. - -**The validator floor is pinned as a set, not a count.** -`tests/test_crate_smoke.py` materializes a real project and runs the -official `rocrate-validator` (dev dependency) against Provenance Run -Crate 0.5: REQUIRED must be clean, and RECOMMENDED failures must stay -within `_FLOOR` — checks the crate cannot truthfully satisfy (the -workflow's id is its in-crate path and the shape wants `^http`; an -annex-stored image has no registry; lc knows no publisher and no -affiliation). A new failure is a regression, a disappearing one is the -floor to shrink. Gated like the container smoke suite: -`LC_CRATE_TESTS_REQUIRED=1` in CI turns the no-validator skip into a -failure, two tests cover the guard. - -**A foreign write is `stale`, everywhere, and it is history, not -hashing.** The hole layer 4 recorded: a skip returns the *recorded* -digest, so a hand-edited-and-committed output (the agent-forged-file -scenario) reads `current` forever. The check: every output is -committed, so a hand edit requires a commit, and `dataset.last_writer` -names it — an output is cleanly written iff the commit that last -touched its directory carries its own run-record subject. -`materialize.datalad_run_subject` is the one spelling of that subject -(named for whose format it is — datalad's, matched by `rerun`'s regex), -shared by `run_record` (the composer) and `_foreign_write` (the -comparator), because two strings here would drift. A hit is a -*contradiction*, not a circumstance — the manifest no longer describes -the bytes — so it classifies `stale` and **the next run remakes it**, -the same philosophy as the dirty refusal's path split: `results/` is -lc's to write, and a committed hand edit is wreckage with a commit -message. Three coherent surfaces, one walk: `lc status` reports it -stale (still exit 0; `OutputStatus.foreign_write` carries the foreign -commit's sha — what prose cannot give a machine consumer), `--check` -plans it, and materialize's driver answers the history question up -front and hands each worker the offending `dataset.LastWrite` — -workers have no git, the HEAD discipline on a third value. History -enters `assets.classify` as **one more input value**, exactly like -check mode's sentinel: computed by whoever has git, handed in as the -*fact*, and the rule stays pure — the verdict's prose is classify's, -like every other why, so `calls_for_a_remake` remains the one bool -that turns a state into an action, with no `foreign or …` re-spelled -at any call site. `last_writer` answers "cannot say" as -empty, never an error — a read-only verb must not refuse over an -unborn HEAD or a stripped `.git`. A full-rehash `lc verify` was -considered and rejected: O(bytes), blind on unfetched outputs (history -answers on a bytes-free clone), and with no lifecycle moment enforcing -it. What it leaves to existing tools, on purpose: uncommitted edits -are a dirty tree, annex object corruption (the thin-write hazard -included) is `git annex fsck`, and a manifest lc itself mis-recorded -is the accepted residue only a rehash would catch. A commit that -*forges* the run-record subject defeats the check — as a regenerated -hash would defeat a rehash; the threat model is accidental damage and -shortcuts, not adversaries (spec §7's line). - -**Forging in a test must break the hard link first.** Results are -committed thin, so two byte-identical outputs share one annex object — -an in-place `write_text` on one dirties the other, which is the -recorded thin hazard demonstrating itself. `test_materialize._forge` -unlinks before writing; a new tampering test should too. - -## Key Invariants (layer 9) - -**Jupyter is the server; the page is just astra-ui.** The server is a -`LabServerApp` — token auth, the page config, Jupyter's Contents API — -after JupyterLab's `examples/app` (BSD-3, credited where adapted), loading -no other server extension but jupyter-live-content -(`serverapp_config`'s `jpserver_extensions`, which turns extension -discovery off; `labextensions_path = []`) and serving -no settings or themes. The page is -not a JupyterLab front end: React renders astra-ui's `Inventory` straight -into the body, over a `ContentsManager` from `@jupyterlab/services`. -A `JupyterFrontEnd` (shell, plugins, LabShell's menu and tab bars) was -built and dropped: with one view and no picker, its only remaining job -was loading a JupyterLab theme for a page with no JupyterLab interface. -Light or dark follows the OS (`prefers-color-scheme` → -`data-astra-color-scheme`). The frontend depends on seven packages — the six it renders with, plus -`pdfjs-dist` for reading papers; keep it that way. - -**The browser resolves the spec; the server's routes answer only what the -Contents API cannot.** -`@astra-spec/sdk` resolves over a `ProjectReader` on the Contents API, -rooted at the project, and `@astra-spec/ui` renders. Python never parses -the spec for the app — astra-tools has no serializer for the resolved -form, and writing one would be a second resolver. A path is stat'ed from -its parent's listing, and a directory is listed only once its own parent -shows it: resolution asks about every unmaterialized output, and this -answers "absent" without a failing request for each. - -**The page follows the disk, through the ecosystem's own watcher.** -Jupyter has no core push for files changed outside its API (the server's -`contents_service` events cover API calls only; jupyter-collaboration -polls `last_modified` per open document). `jupyter-live-content` -(jupyter-ai-contrib, built for agents rewriting files) is the piece that -does: a server extension — the one `LightconeApp` loads beside itself, -through `serverapp_config["jpserver_extensions"]` — watching with -`watchfiles` and pushing `server_update` over an authenticated WebSocket -to clients that sent `client_opened` for a path. Only its server half is -used: the page speaks the protocol directly. The reader records every -path a resolution asks about, present or not, and the page subscribes to -each with its ancestor directories — the extension watches a subscribed -path's *parent*, so a directory's own creation is what reveals a first -output under a `results/` that did not exist — then re-resolves on any -update (debounced) and reasserts every wanted subscription, including paths -that were already subscribed. This makes newly created parent directories -acquire their own watches. Contents models also carry `watchPaths`: the -resolved file target and its sidecar named by `assets.manifest_path`. The -reader watches these without reading hidden contents. A late manifest write refreshes provenance -independently of the artifact's size and mtime. The contents manager ignores -`require_hash`: live-content otherwise reads entire artifacts to hash them, -but this page needs only change notifications. An open dialog survives it. -A dropped socket — a sleeping laptop, a reset `ssh -L`, an idled-out -server — is reopened with backoff (1 s doubling to 30 s), and while it -is down the page says it is not following the disk and to run `lc view` -again if that persists. On reopening it resubscribes and re-resolves -once, because whatever changed meanwhile went unannounced. Reopening -authenticates with the current server's cookie. Every server run -has a fresh in-memory signing secret and a fresh random token independent -of `JUPYTER_TOKEN` and `JUPYTER_TOKEN_FILE`. An expired server therefore -requires a fresh `lc view` link; an old page cannot log into any later -server, even for the same project. This matters on shared hosts: another -user can bind a stopped server's port and collect the old page's cookie, -which must not authenticate a replacement server. The tests replay that -cookie against new servers for both the same and another project. -Plain HTTP on localhost still does not authenticate the endpoint itself: -a port squatter can impersonate the stopped viewer. SSH forwarding to a TCP -port does not remove that boundary; fresh credentials prevent cross-run -replay, not endpoint impersonation. -`tests/test_serve.py` pins the push against a real server. Residue: -`watchfiles` uses inotify, which does not see writes made from another -host on a network filesystem — `WATCHFILES_FORCE_POLLING=1` is its own -switch; lc sets nothing. The protocol is the extension's, not a published -API, so it is pinned below 0.2. - -**The baseline universe, astra-ui as released, records in its dialogs.** -The page resolves `universes/baseline.yaml` — a project without one -shows the SDK's own error rather than a guess at another. Each record opens in `Inventory`'s own dialog, and a deep link -(`?record=`) is that dialog open at load (`defaultDetail`). Records as -dock tabs were built and dropped to stay on the released -`@astra-spec/ui`. - -**astra-ui's host slots are filled from project files, or through the -app's own routes (`app/routes.py`) where the Contents API cannot answer, -or left empty:** - -- `renderArtifact` — astra-ui's own `ArtifactPreview` over bounded reads - through `/files/`: figures as images, CSV/TSV tables from a 64 KiB - ranged sample, JSON tables and metrics under 2 MB. The bounds are - jupyterlab-lightcone's, so an output previews the same in both hosts. - A preview is keyed on its file's cache token and loads once per - version, never on an unrelated change elsewhere on disk. -- `renderProvenance` — `GET /lightcone/api/run?path=`, - answered by lc's manifest parser (`assets.loads`, also used by - `assets.read`). The route opens the sidecar through the same confined - descriptor-based reader as project files. Manifests remain hidden from - the Contents API. Provenance refetches on analysis refresh; artifact - previews retain their separate size/mtime cache keys. -- `paperMetadata` + `loadPdfJs` — cited papers read in place. astra's - paper cache (`~/.cache/astra/papers`, what `astra paper add` fills and - quotes are verified against) lies outside the project: - `GET /lightcone/api/papers?doi=…` answers title, authors and a `pdfUrl`, - and the PDFs themselves are served by Jupyter's own - `AuthenticatedFileHandler` rooted at the cache (`/lightcone/papers/…`), - which brings ranges for pdf.js and 304s on reopening. A cached paper - wins; an arXiv DOI the cache lacks reads from `arxiv.org/pdf/` (the - rule jupyterlab-lightcone and the published site share); any other DOI - keeps the viewer's link to it. A title is whatever the cache holds — - `astra paper add --pdf` records none, `astra paper fetch-metadata --all` - fills them. Papers load after the analysis is shown, and again only - when the cited DOIs change. pdf.js is the legacy 4.8.69 build astra-ui - validates, its worker created through `new Worker(new URL(...))` so - rspack emits it. There is no fetch route: the page never writes the - cache. -- `getOutputStatus` — `GET /lightcone/api/status`, answered by - `materialize.status`, the call `lc status` makes, keyed - `/` with the engine's own `why` as the detail. - Current/behind/stale is `assets.classify`'s answer, and the page holds - no copy of it — asking the engine is how jupyterlab-lightcone fills - the same slot. The report's warnings travel with the verdicts and the - page shows them above the inventory, because an unfetched input's - verdict reads "changed" and only the warning says it is merely not in - this clone — the same two lines `lc status` prints. Asked after every - resolution, inside the resolution loop, since whatever changed on disk - may have changed it and the walk hashes every declared input: one walk - at a time, a change during one asks for one more. A project the engine - cannot read is a 422 and the page shows no state rather than an error. -- **Left empty, deliberately:** `renderCodeLink` (no editor to open - in), `onFetchPaper` (it writes astra's cache, from a click). - -**Every route needs the token** (2026-10, with the routes, reversing "the -server adds no routes"). jupyter_server still serves a handler that -declares no auth rule to anyone, so the server runs with -`allow_unauthenticated_access=False`, turning a missing rule — ours or an -extension's — into a refusal; the routes also carry `@web.authenticated`, -which jupyter_server expects of every extension handler. -`test_every_route_needs_the_token` fails only with both removed. The -routes are prepended: jupyterlab_server ends its own list with a -`/lightcone/.*` catch-all that answers with the page. - -**The token reads; it never writes or runs code** (2026-10). The URL -`lc view` prints ends up in agent transcripts, logs and `ssh -L` -tunnels, and a stock jupyter_server token is a shell: kernels and -sessions run code, `/api/config` writes `~/.jupyter`, and the Contents -API writes and deletes project files. So the server is -`LightconeServer`, a `ServerApp` whose `default_services` (jupyter's -documented way to expose a subset) keeps only `contents` and `files`, -and its `authorizer_class` is `ReadOnly`, which allows the `read` action -and nothing else. Every Jupyter handler that changes something is -`@authorized` and maps POST/PUT/PATCH/DELETE to `write`, so one rule, -asked per request, refuses writes, checkpoints and notebook trust alike -with a 403 — including any write path a later jupyter_server adds. (Overriding -the contents manager's write methods was the first version: it guarded -one service, by a list of upstream internals.) Terminals are absent -already: jupyter_server 2 ships them as an extension, and discovery is -off. `test_the_token_reads_and_runs_nothing` pins both halves, each -mutation-checked. Kernels being gone also keeps `shutdown_no_activity` -honest, since a live kernel or terminal holds the server up. - -Three writes happen outside the authorizer, and each is switched off by -the class Jupyter lets the app name (found by probing, 2026-10): -- **The login form's password change.** A POST to `/login` with the token - as `password` and a `new_password` makes jupyter write a hashed - password into `jupyter_server_config.json` and accept it from then on. - (Under `JUPYTER_NO_CONFIG` that file lands in a throwaway temp dir, not - `~/.jupyter`, but the password still works on this server.) - `TokenLogin` redefines `allow_password_change` as `False` and *not* - configurable. -- **The notebook trust store.** Any Contents API read of an `.ipynb` - checks its signature, which creates `nbsignatures.db` and - `notebook_secret` in the user's Jupyter data dir. `ProjectContents` - gives the contents manager a notary with an in-memory database and a - random secret (nbformat's documented `:memory:`), so every notebook - reads as untrusted, which nothing here acts on. -- **Corrupt-notebook recovery.** Jupyter's atomic-writing mode can rename - an intermediate over a corrupt notebook during GET. `ProjectContents` - disables that mode so a failed read cannot mutate the project. - -The WebSocket was checked too: jupyter-live-content accepts only -`client_opened`/`client_closed`. What the server still writes is its own -state: the runtime file and log under -`~/.lightcone/serve//`, and, as with any `JUPYTER_NO_CONFIG` -server, an empty `jupyter-clean-cfg-*` temp dir jupyter_core never -removes. - -**File reads stay beneath their serving root.** `app/files.py` confines -both Contents API reads and ranged static-file reads; the paper cache uses -the same handler with its own root. External symlinks and aliases to hidden -files are refused. Internal links work, including locked git-annex files -pointing into `.git/annex/objects`. After resolution, directory-relative -opens with `O_NOFOLLOW` anchor every step, preventing a concurrent writer -from replacing a checked component with an escaping symlink. Only regular -files are read; a planted FIFO cannot hang the server. Directory listings -omit unavailable children rather than failing the whole listing. - -**Distribution builds always rebuild checkout frontend inputs.** The native -Hatch hook (`hatch_build.py`) runs `npm ci` and `npm run build` when `ui/` -is present. No timestamp heuristic can miss styles, favicons, locks, or -compiler configuration. The sdist includes the hook and built assets but -no `ui/`; its wheel build verifies the assets without looking for npm. -Editable Python installs skip frontend building; build it explicitly for -`lc view`. Browser regressions use Playwright's Chromium, installed with -`python -m playwright install chromium`; `LC_UI_TESTS_REQUIRED=1` makes -missing browser/build prerequisites fail instead of skip. - -**The page is versioned by its bundle's bytes.** Jupyter serves the -static directory as immutable by path, ignoring `?v=`, so `?v=` is the -only thing that makes a browser fetch a rebuilt `bundle.js` — and it is -a hash of that file, not the package version, which a development build -does not move. Lazy chunks (pdf.js and its worker) are named by content -hash for the same reason. - -**The verb names the intent, the module the mechanism.** `lc view` is -what an agent runs to show the user the analysis (2026-10, renamed from -`lc serve`); `engine/serve.py`, `tests/test_serve.py` and -`~/.lightcone/serve/` keep the server's name, because they are the -Jupyter server and nothing else. - -**`lc view` is detached, idempotent and stops itself.** It prints the -URL of this project's running server, or starts one and prints that, -and returns — an agent's call must not block. Discovery is jupyter's own -(`list_running_servers` over the runtime files, matched on `root_dir`); -the idle stop is jupyter's own (`shutdown_no_activity_timeout`, 30 min). -The page talks to the server only while it loads — it does not poll — so -the clock runs from the last page load, open tab or not; an agent asked -again simply starts a new server. The -server is spawned with `python -I` and `JUPYTER_NO_CONFIG=1`, so neither -project-local Python modules, `PYTHONPATH`, nor a user's Jupyter config -can change what an agent gets back. -It never opens a browser. `RECORD` is passed through to the URL -unchecked: validating it would need the resolver this layer refuses to -duplicate, so a record the analysis does not have opens no dialog. - -**Serve state is private to the user and kept per host, never in the -project.** Runtime files and the log carry the token, and a project on -group-shared storage would hand it to every collaborator; so they live -in `~/.lightcone/serve//`, mode 0700, log 0600. Per host -because NERSC-style sites share `$HOME` across login nodes while jupyter -can only tell whether a pid is alive on its own node — shared, a call on -one node would delete a live server's file from another, or trust a -reused pid. A call on another node starts its own server; the old one -idles out. The URL is `localhost` on the named host (stderr, and -`--json`'s `host`): from elsewhere, `ssh -L` to that node. Accepted -residue: two concurrent first calls can start two servers. - -### Recorded decisions - -- **Offers name their provider; there are no connections (2026-10).** A - catalog used to be `connections` — a UUID `namespace`, a `provider`, a - `context`, `launch` settings — plus offers referencing one by name. lc runs - inside one Slurm environment, so `context` (`--clusters`) went, and with it - every provider reaches exactly one native authority: this host, or the - cluster the Slurm client reaches by default. That left the connection a pure - indirection. Offers carry `provider`; worker launch settings joined each - offer's `config` (`task_slots_per_node` was already validated per offer); - `connection_root`, the one setting read outside `plan`, is the catalog's one - top-level key; and the cluster ID encodes the provider instead of the - namespace, so a full ID routes with no catalog lookup. The UUID was worse - than redundant: Slurm discovery never filtered on it, so two catalogs with - different UUIDs gave one job two IDs. What a full ID does *not* free from - the catalog is connection material, which lives under the launching - catalog's `connection_root`, for Slurm as for local: native inspection and - `down` work from any catalog, connecting does not. Launch creates the - submission's directory (`/slurm/`, its log and each - attempt's TLS material) before submitting, so a running job without one - is refused as launched under another root rather than told to wait. - `PROVIDERS` lives beside the `Provider` protocol in `model.py`, and an - offer naming anything else is a catalog error — a typo once loaded and - then failed every discovery, blocking every launch. Discovery queries `Catalog.providers` — - those the offers name, plus local always — so removing a provider's last - offer hides its jobs from name lookup and the listing, never from a full - ID. Re-add a per-authority concept only with a venue that genuinely has two - authorities of one provider. - -- **Guard NERSC login nodes by default (2026-09).** This reverses the earlier - no-login-node-guard decision: an unconfigured first launch must not allocate a - whole shared login node. Detect the documented NERSC environment marker and - login hostname together, without DNS queries or scheduler probes. A runtime - restriction also covers copied or incomplete catalogs; no generated file or - override flag is needed. Local compute in interactive compute-node sessions - remains available, subject to the configured local policy. - -- **Local compute ends when idle, not at a fixed age (2026-09, issue #233).** - An offer's `time` is `{default?, max?, idle?}` and needs a `default` or an - `idle`; the built-in local offer is `{idle: 30m}`, so a long recipe finishes - and the cluster stops 30 minutes after the last task. `idle` is handed to the - scheduler as Dask's own `idle_timeout` — its activity test (running, queued or - unrunnable tasks and any transition reset it; clients and `scheduler_info` - polls do not, measured) rather than a tracker of ours. `--time` stays a hard - walltime (SIGALRM, unconditional on activity); with both, the first to fire - ends the allocation, and the built-in offer has no `max` because a ceiling on - `--time` is meaningless when omitting it means unbounded. When the scheduler - closes without the owner asking (no SIGTERM yet), a `SchedulerPlugin.close` - hook records the reason in `error.json` and SIGKILLs the session itself, like - the walltime path — from the hook, so a scheduler that idles out before - `LocalCluster(...)` returns still ends the owner, and never through - `LocalCluster`'s own close, which waits ~34 s on the departed scheduler - (measured), holding the one-per-machine slot and the name. `SCHEDULER_CONFIG` - pins `idle-timeout: None`, so only an offer sets one — ambient Dask config - reaches neither local nor Slurm schedulers. `compute.connect` (execution only; - `status` connects through the provider) submits one no-op task, so a - driver's preparation — annex fetch, image build, sync — starts with a full - countdown. Slurm refuses `time.idle` and still needs `time.default`: its - allocations end at the native walltime, and an ignored idle timeout would be - a lie. Accepted residue: a preparation longer than the timeout still loses - the cluster; a driver pausing between tasks (a long annex commit) counts as - idle; and without `--time` nothing bounds an owner whose scheduler loop - wedges — the walltime's SIGALRM was that bound, and a second timer only for - it was judged not worth its code. - -- **Local compute accompanies remote catalogs (2026-09).** The built-in offer - uses the host's usable CPU/RAM capacity and follows configured offers, replacing - the previous one-CPU/1-GiB fallback that disappeared when a catalog existed. - A catalog's own local offers replace it, including to set a smaller budget - (a `local.resources`/`local.time` block once did that too, a second way to - say what an offer already says, so it went). Login-node catalogs disable - local launch and execution through `allow_local: false`. Inspection and - termination stay available. One local - allocation per user per machine is enforced across catalogs and connection - roots by scanning the process table for a live owner before launch - (`local._running_owners`: a session leader of this user running - `_OWNER_ARGS`, the command launch and `_process` share). This replaced an - `flock` held for the owner's lifetime, because NERSC home filesystems do - not support `flock`. Accepted residue: overlapping launches can both start, - and the scan covers one PID namespace, so a container sharing the home - does not see the host's owner. A missing identity record is transient - ("retry shortly"); an unreadable one is not, so that refusal names the PID. - The suite scopes the scan to its own temporary tree - (`local_allocation_scope`), so a developer's running cluster does not refuse - test launches. - -- **The engine is the host's uv tool, never a project dependency** - (2026-08, reversing spec §2's engine-in-lock rule and deleting layer 3). - `lc init` scaffolds no `lightcone-cli` dependency, and there is no - launcher and no delegation: the `lc` that was invoked is the engine that - runs. What the pin bought, and where each guarantee went: - - *The engine inside `env_version`.* Gone, and deliberately: an lc - upgrade now moves nothing — no output ever reads `behind` over an - engine release, completing the reversal that already took - `env_version` out of `definition_version`. The engine is attestation - (`lc_version` in every manifest, `worker.lc_version()`), not identity. - - *Reruns reconstruct the exact engine from the commit's lock.* - Re-provided in the run record itself: `cmd` pins the engine through - `uv run --no-project --with ''` — by version for a - release, by source commit for a dev build (see the layer-4 run-record - invariant). Needs PyPI or the git remote reachable at rerun time - where the lock needed neither; recorded, not hidden. - - *Driver/worker version skew "structurally impossible".* Re-provided - by sharing the **installation** instead of the lock. What a Dask - worker process actually needs is `lightcone.engine` importable at a - matching version — pickle serializes `worker.materialize` by - reference and `Task`/`Versions`/`TaskResult` by class reference. - Pinned by the suite since layer 7 - (`test_a_processes_cluster_fits_through_the_seam`, a full - materialization through a `LocalCluster(processes=True)`): the - code works unchanged across process boundaries, and workers - need **no git and no git-annex** (the driver owns git alone; - `data_version` is pure file hashing). So workers run the driver's own - interpreter (`sys.executable`, the default for both providers), which - on HPC is the tool environment on the shared filesystem: driver and - workers are the identical installation, which also satisfies - `distributed`'s own client/scheduler/worker coherence requirement. In - containerized mode the same host installation runs the workers; only - recipes enter the image. The `assets.Versions` memo is filled on the - driver before tasks are serialized, so a declared input shared by many - tasks is hashed once per run, not once per worker process. - - *The engine's dependency closure left the record entirely, and is - mostly not replaced.* The project lock used to pin what the engine - resolved — most concretely the git-annex build that wrote the bytes. - Now `lc_version` names the engine, the hardening pass added - `uv_version` beside it (the one closure member that decides which - artifacts a lock installs), and nothing pins the rest. Accepted - rather than re-provided: the alternative is hashing an - environment lc does not own into artifacts it does, which is the - over-sensitivity the `behind` model exists to avoid — both fields - are attestation, never identity. - - *Layer 6 resolved its note the other way:* the image gets **no - engine layer at all**. The engine stays on the host in containerized - mode too — the container is the recipe's world, never the engine's — - so the engine version is not a tag input and an lc upgrade still - rebuilds nothing. See the layer-6 invariants. - What it buys: a project's resolution is freed from the engine's own - pins (no astra-tools/click/rich/distributed/git-annex in any project - lock); the git-annex wheel floor gates only the tool install; the eval - workflow exercises the branch under test rather than a released engine - resolved from PyPI; and recipes no longer find `lc` or `git-annex` on - the sandbox PATH — the project `.venv/bin` no longer carries them, - which is the boundary telling the truth. The `UV_*` ambient scrub the - launcher would have done landed in the hardening pass, in - `project.child_env` — it protects `env_version`'s install-settings - term, not the delegation that is gone (see the layer-2 residue note - for the allowlist). -- **Reads stay restricted, and the OS baseline is ours to maintain.** - Codex restricts reads too now, but its Linux read baseline is a *mount - table*, not a path list — there is nothing to adopt there. Keeping the - FHS allowlist is a deliberate choice about which way the failure - points: a missing allowlist entry is a **loud** `EACCES` that gets - reported and fixed once for everyone, while any exclusion-based scheme - fails **silently**, in exactly the thing the layer exists to prevent. - Worth remembering when the list next comes up short: none of the three - bugs we hit (ELF loader, interpreter root, `/dev/urandom`) came from - the list — two were derived paths and one was already in it. -- **Devices, `/proc`, and `/sys` are granted generously, on purpose.** - Writable: the terminal set (`/dev/tty`, `/dev/pts`, `/dev/ptmx`), the - discard devices (`/dev/null`, `/dev/zero`, `/dev/full`), and `/proc` - and `/sys` whole. The threat model is accidental leakage, not a hostile - recipe (spec §7), and **none of these is a channel undeclared *inputs* - arrive through** — that is the test to apply when the next one comes - up. Tightening them buys nothing and costs real failures: - `pty.openpty()`, `/dev/full` ENOSPC handling, `oom_score_adj`, MPI and - CUDA runtimes poking `/sys`. - It also reads more permissive than it is — Landlock only ever *removes* - access, never adds it, so ordinary Unix permissions remain the real - gate (devpts hands each pty to its allocating user at 0620; nearly all - of `/proc` and `/sys` is root-owned). Granting them just stops lc - adding a second, more confusing denial on top of the OS's own. - The line is drawn at `/dev/urandom` and `/dev/random`, which stay - read-only: writing those seeds the *host's* pool — a side effect on - the machine rather than on the run. - The general rule, learned here: **a grant whose absence produces an - unattributable error is one to make freely.** Without devpts, - `pty.openpty()` raises `OSError: out of pty devices` — naming neither - a path nor the sandbox, so `denial.explain()` can extract nothing and - only the trailer fires. A denial the user cannot act on is worse than - the access it withheld. -- **Landlock stays the Linux mechanism; bubblewrap is not adopted.** - Codex moved its Linux default to bwrap+seccomp (Landlock is now - `--use-legacy-landlock`) because it needs rights *subtraction*: - `read_only_subpaths` inside a writable root, to stop an agent writing - `.git/hooks` and escalating. Landlock cannot express that, and their - own older code silently dropped the carve-out on Linux as a result. - **That requirement is not ours** — spec §7's threat model is accidental - leakage, not a hostile recipe, and our policy's exceptions always - *widen* (a writable output dir inside a readable tree), which is - exactly the direction union semantics handle for free. The cost would - be real: a bundled `bwrap` binary, a user-namespace probe, a WSL1 - refusal, and the Ubuntu 24.04 AppArmor wall that made - `hermeticity-enforcement.md` §3 call bwrap "an opportunistic upgrade, - never the requirement". Re-add triggers: a policy shape that genuinely - needs subtraction, or ambient `bwrap` becoming universal. -- **The ASTRA `container:` directive is ignored, entirely.** A spec may - carry one (astra's scaffold no longer writes it), and lightcone-cli does - nothing with it: not read, not stripped, not validated, not migrated. - The environment is `pyproject.toml` + `uv.lock` (spec §2), so such a - project simply carries a key no code path consults. Don't "fix" this by reconciling the two — a later layer will - decide whether the key is dropped upstream, refused, or migrated. -- **Multi-runtime, podman recommended** (2026-08, layer 6 — superseding - an earlier "podman only" plan decision). podman and docker ship - tested; the backend seam and the `docker-archive` format are chosen so - apptainer/singularity become thin layer-7 backends over the same - archive (they exec `docker-archive:` content directly, and HPC hosts - that cannot build obtain images through the annex). Build-capable - (podman|docker) and run-capable (all four, eventually) are therefore - separate questions in `container.py`. docker's daemon is probed at - detection — a CLI with the daemon down is the common broken state. -- **Bare-recipe run records: considered and rejected** (2026-08). - datalad expands container templates at *record* time and `rerun` - executes the literal string — datalad-container has no rerun hook — so - a record whose `cmd` is the recipe replays on the host, unconfined, - with no manifest, and a containerized spelling would freeze one - runtime's flags into git forever. The worker-as-executor keeps the - record declarative and runtime-neutral; mechanical no-lc replay is - layer 8's WRROC export, for which the manifest + archive now carry - everything. -- **Single-arch archives, recorded residue** (layer 6 → layer 7). The - committed archive is the architecture it was built for, and the - manifest says which. An Apple-silicon build cannot serve an amd64 - venue; solving that (arch-suffixed archive paths, `--platform` - cross-builds) is layer 7's, with the venue that makes it real. Old - archives accumulate in annex history; reclaiming them is the user's - `git annex unused`/`drop` — no GC verb. -- **The hardening pass** (2026-08, post-layer-8) closed the residues that - did not need Perlmutter, and re-examined two it then deliberately left: - - *A recipe is narrowed to its own output directory* — reversing the - layer-5 "not narrowed" decision. The old rationale rejected the - sandbox as a second *integrity* mechanism, and that half stands - (`data_version` remains the only answer to "are these bytes what - produced them"); what the narrowing is instead is leak *prevention*, - closing the cross-write residue, whose corruption was undetectable by - construction. A probe keeps `results/` whole (no output id), the one - probe/recipe asymmetry. - - *The network is uncontrolled everywhere*: the OCI wrap dropped - `--network none`, its one restriction, so all three mechanisms attest - `network: allowed` symmetrically and no consumer can read a promise - into "containerized". `denied` stays in the `Attestation` type for a - mechanism that genuinely emits a denial flag. - - *The forged run-record subject stays a recorded residue*, re-examined - and left: the threat model is accidental damage and shortcuts, and a - copied `[DATALAD RUNCMD]` subject is already deliberate — a - body-verifying comparator raises the forgery cost without changing - who it stops, and nothing history-based is adversary-proof anyway. - - *The crate validator floor stays pinned at five*: three entries are - publisher/affiliation metadata lc genuinely does not know, and a - `[tool.lightcone.publication]` surface was considered and rejected — - revisit when a real deposit target demands it. - - *lc's commits are partial commits, and a frozen execution worktree - is deferred, not rejected.* `dataset.save` commits with the same - pathspec it stages, so each save is built from HEAD plus its own - paths alone — work the user staged while a graph ran stays staged - and is named by the end-of-run warning, never swept into an output - or crate commit. The stronger move — executing in a dedicated - `git worktree` at the starting commit, which would also freeze the - *code* a long run reads — was considered and deferred to the venue - era: the branch dance under a live checkout, result propagation - back into it, and a second environment/mid-run-gate story outweigh - a hazard the warning now names precisely, and the case that makes - it genuinely worth it (editing on a login node while an `sbatch` - materialize runs for hours) arrives with the submission-model - venue. - -### Recorded decision: one output, one file (post-hardening) - -`Output.format` landed in astra-spec 0.0.14 / astra-tools 0.2.17, and lc -took it as the missing half of a long-standing asymmetry: `Input.source` -always declared a path with an extension, while an output declared -nothing and `{output}` was documented as "the path the artifact will be -written to" — a path the schema never defined. What changed, and why: - -- **An output is one file, not a directory.** The multiplicity was born - the moment a recipe was handed a directory and picked a name inside it. - There is no directory now, so the natural idiom (`savefig(args.out)`) - produces exactly one file at exactly the declared path. The cost, - stated: a figure plus its data, a zarr store, a checkpoint directory - must become separate outputs or a container format (`.h5`, `.zip`, - `.tar.gz`). No flag re-opens it. -- **The filename contract between producer and consumer is gone.** - `{inputs.X}` renders to the upstream's *file*, so nothing has to know - what is inside a directory it was handed. -- **`.gitattributes` is convergence's to get right, not a run's.** - `dataset.save` passes `annex.dotfiles=true`, so the sidecar's leading - dot decides nothing and the attributes decide everything — annexed, a - run is green locally while every clone reports the project as never - materialized. A `check-attr` preflight in `materialize` was written and - removed: `lc init` owns that file, its repair appends the exemption and - its disorder guard blocks a file it cannot fix, and a second answer in - the run path is one more place for the two to disagree. The residue, - stated: a project scaffolded before the exemption existed annexes its - manifests until someone converges it — which this change already - requires, since every output needs a `format:` added by hand. -- **The orphan walk is a set difference, never a reconstruction.** - Expected sidecars come from `Task.manifest_path`; present ones from - `git ls-files`, which still finds a manifest whose output the spec has - just dropped — the edit that orphans it is the same edit that would - drop it from anything derived from the graph. -- **No migration, by decision.** Nothing looks for `.lightcone-manifest.json` - and nothing detects directory-shaped outputs from a previous engine; - they are invisible to every walk. `git rm -r results/` and - re-materialize. -- **A nested spec is not buildable.** lc materializes a flat analysis: - ASTRA qualifies an output declared in a sub-analysis as `.`, and - `output_path` refuses a dotted id because the sidecar could not be told - from the payload. Supporting nesting means deciding where a nested - output's file goes, which is its own change (issues #201, #202). - -### Recorded deviations from the spec - -- **No `AGENTS.md` scaffolding** (spec §2 calls for an agent notes - stanza). Dropped by decision: it documented four verbs, three of which - don't exist yet, and the scaffold shouldn't assert behavior the CLI - can't deliver. Revisit at layer 4, when the verbs it describes are - real. `tests/test_project.py::test_converge_writes_no_agent_notes` - pins the absence, so re-adding it is a deliberate act. -- Scaffolded file bodies stay descriptive of what works today — see the - trimmed `results/README.md` for the same reason. -- **The Landlock policy travels as JSON on argv, not as an inherited - ruleset FD** (spec §7 specifies the FD, built before fork and passed - with `pass_fds`). The shim builds and applies the ruleset itself, which - is also what the codex CLI does. This makes `wrap()` a pure function and - closes §11's own blocking spike — *"does the Landlock FD survive - `uv run`'s spawn/exec chain? an FD cannot be reopened"* — by making the - question moot. It also survives into a container later, where a host FD - cannot. The document is deliberately *not* versioned: the wrap always - invokes the shim on lc's own interpreter, so writer and reader are the - same lightcone-cli by construction, and a compatibility field would be - backward-compat machinery with no consumer. -- **Network is not controlled, on any mechanism**, by decision. §7's - matrix has Seatbelt record `denied`; the generated SBPL explicitly - allows network, and since the hardening pass the OCI backend emits no - `--network` flag either (it briefly shipped `--network none`, dropped - for consistency: three mechanisms, one answer). Every mechanism - attests `network: allowed` — symmetric and honest, nothing pretends - to a control it does not apply. (codex ships a seccomp denylist for - this; adding one is a live option, not a gap we are hiding — and it, - or a runtime that genuinely denies, is what the `denied` literal in - `Attestation` is reserved for.) -- **`lc run` has no rename guard and no sandbox flags.** §4's guard - against `lc run ` existed only for muscle memory from the - pre-rebuild CLI — backward compatibility we do not promise — and §7's - `--require-sandbox` / `--no-sandbox` / `--sandbox-debug` are all - absent: there is no hatch to escape the sandbox, so there is nothing - for the flags to switch. The verb takes a command and nothing else. -- **The denial's remedies are only what works today** — `uv add` for a - Python package, the system layer (`apt-install` + the containerize - note, real since layer 6), the ASTRA input declaration for data, and - "a recipe writes only its own output directory; a probe writes - `results/`" plus `tempfile.mkdtemp()` for a write - denial. Nothing in a denial message names a verb, flag, or declaration - that does not exist. -- **`Attestation` has no serializer of its own.** An earlier draft - carried a `to_manifest()` with no caller — deleted, because "no dead - code" applies to this layer's own conveniences too. It is persisted by - the layer that needs it: the worker writes `asdict(outcome.attestation)` - into the manifest's `hermeticity` field. `lc run` is still a probe with - no output (§4), so there the attestation is returned and printed, never - persisted. -- **The in-tree write scope is writable; the rest of the tree is not**, - where §4 gives a probe no output and therefore no in-tree write scope - at all. A probe writes `results/` whole; a recipe, since the hardening - pass, writes only its own output directory (the cross-write closure — - see the layer-4 policy invariant and Recorded decisions) — and the - environment a run starts with is the one it finishes with. - - **The shape was chosen because all three mechanisms express it - natively.** A writable directory *nested inside* a read-only tree is - the widening direction: Landlock unions rights over ancestors, SBPL - restates the write tier after the guard, and podman mounts the - project `:ro` with the write scope `:rw` over it — all verified by - running them, and the narrowing to one output directory is the same - shape one level deeper. The reverse — a writable tree with `.venv` - carved out — needs - rights *subtraction*, which podman and SBPL can do and **Landlock - cannot at all**. That asymmetry is the whole argument: the read-only - shape is the only one direct mode and containerized mode can both - deliver, so it is the only one that gives a single UX. - - This was briefly reversed (PR #174) on the premise that "a container - bind-mounts the working tree read-write". True of the default, not of - the mount table we will write — and the experiment that settled it is - two `podman -v` flags. Don't re-derive it from the default again. - - **`_exec_set` grants no directory except the interpreter's own - install tree** (see the layer-5 EXECUTE invariant) — and never - `.venv/bin`. A directory grant is a grant on whatever the directory - holds *later*, which was a live hole for as long as the tree was - writable: `cp /usr/bin/git .venv/bin/` ran a tool the allowlist - denies by name. Belt-and-braces under a read-only tree, one scandir, - keep it. - - **A write-denial test must target a path the OS would let you - write.** `printf x > /etc/…` passes with no sandbox at all — the OS - refuses it for any non-root user — so it pins nothing. The sound - target is a user-owned path that the *policy* makes read-only: a - declared input. Mutation-check every denial test by running the same - command through `Unavailable()` and confirming it succeeds. - - **Enforcement fixtures must not live under `/tmp`.** It is in the - write baseline, so anything pytest's `tmp_path` hands you is - *granted*. The old denial tests passed only because a project under - `/tmp` used to drop `/tmp` from the policy; with that gone they went - green while testing nothing. `tests/test_sandbox_enforcement.py`'s - `outside` fixture is rooted at `$HOME`, which is outside every grant - by construction. -- **`/run` is granted whole**, where codex names only - `/run/current-system/sw`. It reaches `/run/user/$UID` — dconf, the - gnupg and keyring sockets, portal state. Kept deliberately: the test - is whether undeclared *inputs* arrive through a path, and runtime - sockets are not something a recipe accidentally reads data from. - `/etc/resolv.conf` is a symlink into `/run` wherever systemd-resolved - is in use, so the grant also keeps DNS working. - -## Extending the Codebase - -| To... | Read | Key patterns | +runner. + +- Python work never needs Node: an editable install skips the front-end + build. `lc view` and the UI tests do need it (plus + `python -m playwright install chromium` for `test_view_browser.py`). +- Several suites **skip** when their mechanism is absent and CI turns the + skip into a failure via env vars: `LC_SANDBOX_TESTS_REQUIRED`, + `LC_CONTAINER_TESTS_REQUIRED`, `LC_CRATE_TESTS_REQUIRED`, + `LC_UI_TESTS_REQUIRED`. A local green run may have skipped them. +- CI runs Linux (py 3.11–3.13) **and** macOS: the sandbox enforcement + suite is one set of tests over Landlock and Seatbelt, so never assert + one mechanism's or one platform's wording. +- The `uv-floor` CI job runs the uv-touching suites under uv 0.7.0, the + floor in `templates/files/pyproject.toml.tmpl`. Using a newer uv flag + means raising both together. +- PRs need a DCO sign-off (`dco.yml`). + +## Rules (each was asked for in review; none is optional) + +- **No backward-compatibility code.** No guards, versions or migrations + for older CLIs, formats or habits. +- **No foreshadowing, no dead code.** Nothing — code, comment, flag, + message — mentions a feature that does not exist yet; nothing lands + without a caller. `lc --help` advertises only verbs that work. +- **No escape hatches around guarantees.** Enforcement ships without a + flag to turn it off (there is no sandbox opt-out, no `--jobs`, no + skip-rebuild flag). +- **Literal behavior over convenience.** The current directory is the + project root; no walk-up discovery; error rather than guess. +- **Streamline.** No small helpers or rendering layers where a few inline + lines read fine. Don't add a flag whose only user is a test — stub the + seam instead. +- **Never reference a design spec, section number or history in code or + comments.** Comments carry *why*, not how the code came to be. +- **Leave working files alone**; don't touch files merely adjacent to a + change. +- **Honest provenance.** Adapted third-party material is marked "inspired + from", with its license (the vendored `.sbpl` profiles, the app after + JupyterLab's `examples/app`). +- **A dependency enters `pyproject.toml` with the code that needs it.** +- Every change ships tests. When adding a *denial* test, mutation-check + it: the same command through `Unavailable()` must succeed, and the + target must be a path the OS itself would allow (not `/etc`, not under + `/tmp`, which the policy grants). + +Conventions: ruff (E, F, I, N, W, UP; line length 100; py311), mypy +strict with namespace packages, Google-style docstrings on public +functions (click callbacks excepted — their docstring is the `--help` +text). + +## Architecture + +### Packaging + +`lightcone.*` is a PEP 420 namespace: **`src/lightcone/` must never get +an `__init__.py`**. The engine is the host's `uv tool` — never a +dependency of the projects it runs, never inside a container image. +`[project.scripts]` re-declares the git-annex wheel's four entry points +verbatim (a test pins them against the wheel) so `uv tool install` puts +git-annex on PATH. + +### CLI / engine split + +`cli/` owns flags, rendering and exit codes only. `engine/` never imports +click and never prints; it raises `ProjectError`, which the CLI turns +into a clean error. Engine imports happen *inside* command callbacks so +`lc --help` stays cheap. Two modules are `python -m` entry points that +must stay import-light and must never become verbs or console scripts: +`engine/worker.py` (no click/rich) and `_sandbox_exec.py` (stdlib only, +zero lightcone imports). + +Every external tool call (uv, git, git-annex, container runtimes) goes +through one seam, `project._run`, which the autouse `tools` fixture in +`tests/conftest.py` stubs. The `real_tools` fixture opts back out; +storage, worker and materialize tests use real tools on purpose. + +### What a project is (`engine/project.py`, `templates/`) + +A project is `pyproject.toml` + `uv.lock` + `.python-version` in a git + +git-annex repository; uv is the only environment substrate. `lc init` is +*convergence*, not scaffolding: every item goes through `_Converger` +(create if missing, append-only repair, never overwrite user files), and +`--check` is the same decision path with writes off. Templates are real +files using `string.Template`; `.gitignore`/`.gitattributes` converge +entry-wise, and new managed `.gitattributes` lines are always appended at +the *end* of the template. + +Mode is derived, never configured: a `[tool.lightcone.image]` table means +containerized; `[project].license` turns on RO-Crate maintenance. + +### Storage (`engine/dataset.py`) + +Git carries pointers, records and history; git-annex carries bytes, via +`* filter=annex` so the user's plain `git add` does the right thing. +ASTRA's records (`.astra/records/`) stay in git so a bytes-free clone can +still classify everything. Unfetched content +*exists* as a pointer file or dangling symlink — hashing must refuse both +(`ContentNotFetchedError`). Only the driver writes to git; a worker reads +it only through ASTRA's checker. + +### Identity and invalidation (`identity.py`, `assets.py`) + +- Whether a result is still what the analysis asks for is ASTRA's + checker's answer (`assets.check` → `astra.api.check`: `missing`, + `stale`, `unchanged`), never re-derived here. It compares the analysis + now with the analysis at the commit the record names. +- `env_version` = lock ‖ `.python-version` ‖ install settings ‖ image + document — what a result ran under. A record names it as the IRI + `urn:lightcone:env:` (`identity.environment`); ASTRA + records it and never compares it. `identity.SCHEME` is the + `astra.environments` entry point that tells `astra check` about it. +- A result's record holds the commit, input hashes, environment and + runner; its schema is closed, and it is the only file lc writes beside + a result. Nothing the record cannot hold is kept — by decision; don't + add a sidecar. + +`assets.classify` is the single rule, with three states: `stale` (ASTRA +finds the record missing or stale, or the files were last committed by +something other than the result's own run → remade), `behind` (only the +environment moved → reported, left alone unless `--refresh`), `current`. +`Verdict.calls_for_a_remake` is the only place a state becomes an action. +The driver asks once before a run; a task downstream of one that may be +remade asks again in the worker once its upstreams have landed, which is +how a byte-identical rebuild stops the cascade. A committed hand edit of +a result is detected by git history (`dataset.last_writer` vs. +`materialize.datalad_run_subject`) and classifies `stale`. Editing source +code does not invalidate outputs unless the file is a declared ASTRA +input — by decision; don't add heuristics. + +### Execution (`plan.py` → `materialize.py` → `worker.py`) + +`plan.build` asks `astra.api.plan` and carries each planned result across +as a picklable `Task`, keyed by its address (`fit#baseline`, `fit` for an +output built once, `stage.fit#u1` for a driven declared analysis, +`stage/fit#v1` for a result that belongs to one). Paths are measured from +the project root, which may sit below the repository's top. A result is +exactly one file at the place ASTRA gives it +(`results//.`), its record at +`.astra/records//.json`; the recipe runs in its own +analysis's folder. Results of an analysis declared with a `commit` are +refused, not built. `materialize` is the +driver: refuses a dirty tree, converges the environment, submits tasks to +a **borrowed** Dask cluster (dependency order comes from futures as +arguments), and commits each result serially with a datalad-compatible +run record whose `cmd` re-runs the engine-pinned worker. The worker never +raises — it returns `ok/current/behind/failed/blocked` — and the tree +ends as clean as it started (`save` on success, `restore` on failure). +`lc status` reports and always exits 0; `lc materialize --check` gates. +Read-only verbs must never traceback, fetch, or need a cluster. + +### Sandbox (`engine/sandbox/`) + +Every mechanism is a pure `Backend.wrap(policy, argv) -> argv`, so all +backends are testable on any host. `detect()` holds the only platform +branch. `Policy` (what to enforce), `Capability` (what the host can do) +and `Attestation` (what was actually enforced) stay distinct types. The project tree is read-only; a +recipe may write only its output's directory, a probe (`lc run`) only +`results/`. Execute rights are per-file — never grant EXECUTE on a +directory that could be a system prefix. Landlock unions rights while +SBPL is last-match-wins, so rule order matters on macOS only. Network is +uncontrolled on every mechanism and attested as such. Containerized mode +(`image.py` pure, `container.py` impure, `sandbox/oci.py`) uses the mount +table as the mechanism; the image is the system layer only, archived as a +docker-archive in the annex under `.datalad/environments/`. + +### Compute (`engine/compute/`, `cli/compute.py`) + +Compute is explicit: `lc compute resources|launch|status|down` manages +local and Slurm allocations through a `Provider`; `lc run` and +`lc materialize` take a cluster name/ID as first argument and never +create compute. No lc server, registry or custom Dask worker — native +jobs/processes are the authority, names are native labels. One Pydantic +model family (`compute/model.py`) is shared by catalog and providers. +Recipe resources (`execution_resources.py`) use stock Dask resource +admission. Slurm is only simulated in tests; the real Perlmutter +deployment test is still pending. + +### `lc view` (`engine/serve.py`, `app/`, `ui/`) + +A detached, idempotent, self-stopping Jupyter server (`LabServerApp`) +whose token can only *read*: no kernels, no writes, every route +authenticated, file reads confined to their root (`app/files.py`). The +page is `@astra-spec/ui` over the Contents API; the spec is resolved **in +the browser** by `@astra-spec/sdk` — Python never parses the spec for the +app. Server routes (`app/routes.py`) answer only what the Contents API +cannot (records, engine status, the paper cache). Server state lives in +`~/.lightcone/serve//`, never in the project. + +## Where to change things + +| To change… | Start in | Tests | |---|---|---| -| Add the next layer | the spec (§11 = the layer ordering) | Land code + tests + deps together; update the layer table above | -| Change what a scaffolded file contains | `src/lightcone/engine/templates/files/` | Edit the `.tmpl`; add new ones to `TEMPLATE_NAMES`, and a renderer only if the file needs a substituted value or a merge policy | -| Add a value to the scaffold | `src/lightcone/engine/templates/__init__.py` | Derive it from the environment or our own metadata before introducing a constant | -| Change what gets converged | `src/lightcone/engine/project.py` + `tests/test_project.py` | `_Converger.item` / `.file`; repairs only ever append | -| Change how a project stores bytes | `src/lightcone/engine/dataset.py` + `templates/files/gitattributes.tmpl` | Every command through `project._run`; test it against a real annex (`real_tools`) | -| Change how an output is identified | `src/lightcone/engine/identity.py` + `tests/test_identity.py` | Sensitivity tests both ways: what must move the hash, and what must not | -| Change what the image is made of | `src/lightcone/engine/image.py` + `tests/test_image.py` | Pure; structure-and-ordering tests, never byte goldens; every declaration key is hashed | -| Change how images are built, stored or entered | `src/lightcone/engine/container.py` (+ `sandbox/oci.py` for the exec argv) + `tests/test_container.py` | Everything through `project._run`; keep `runtime_for_run`'s two strictnesses; runtime differences are spellings inside `OCIBackend`, never new shapes | -| Change when an output is remade | `src/lightcone/engine/assets.py` + `tests/test_assets.py` | One `classify()`, two callers; `--check` differs by one input value, never by logic. Ask first whether the thing that moved *contradicts* the project or is a *circumstance* — the second is `behind`, not `stale` | -| Change how the spec becomes a graph | `src/lightcone/engine/plan.py` + `tests/test_plan.py` | Ask `astra.resolve`; if the answer is missing, the fix is a PR to astra-tools. Anything ambiguous is a `ProjectError`, never a guess | -| Change how a recipe runs | `src/lightcone/engine/worker.py` + `tests/test_worker.py` | Never raises, never writes git; mutation-check every denial test | -| Change what a run commits | `src/lightcone/engine/materialize.py` + `tests/test_materialize.py` | The driver owns git alone; the tree ends as clean as it started | -| Change where a run executes | `src/lightcone/engine/compute/` + `materialize.cluster_for_run` + `tests/test_compute*.py` | Explicit cluster names or IDs; provider-owned native allocation lifecycle; borrowed standard Dask clients | -| Change what the crate says | `src/lightcone/engine/crate.py` + `tests/test_crate.py` | Pure builder: sorted iteration, no clock, git injected as `writer` and the annex key map as `keys`; structure tests, never byte goldens — the one byte-level claim is render-twice-identical. The validator floor lives in `tests/test_crate_smoke.py::_FLOOR` | -| Change how a foreign write is detected | `dataset.last_writer` + `materialize._foreign_write` + `tests/test_dataset.py` | History, never hashing; `datalad_run_subject` is the one spelling of the record's subject; a foreign write classifies `stale` in every verb | -| Change what `lc view` shows | `ui/src/index.tsx` (+ `@astra-spec/ui` for anything a record looks like) | Released astra-ui only; the spec is resolved in the browser by the SDK, never in Python | -| Add a CLI verb | `src/lightcone/cli/commands.py` | `@main.command()`; keep logic in the engine, raise `ProjectError`, render here | -| Add a sandbox mechanism | `src/lightcone/engine/sandbox/` | One module with a `Backend` (`wrap` pure, `attest` honest) + one line in `detect()`. Nothing above the seam changes | -| Change what a sandboxed command may touch | `sandbox/policy.py` + `tests/test_sandbox_policy.py` | Path sets only — no mechanism ever leaks in here | -| Change a denial message | `sandbox/denial.py` + `tests/test_sandbox_denial.py` | Remedies must be copy-pasteable and real *today*; the trailer stays unconditional | - -## Test Patterns - -- `tests/conftest.py` — the `tools` autouse fixture stubs - `engine.project._run`, emulating each tool's observable effect (`uv lock` - writes `uv.lock`, `uv sync` makes `.venv`, `git init` makes `.git`, - `git annex init` marks the repository annexed) and recording every argv. - `uv_calls(tools)` narrows it to uv; `probes(calls)` to the read-only - `--check` probes. Under the stub, tests are hermetic: no network, no - resolution, no subprocesses. The `real_tools` fixture opts back out, - putting the real `_run` back — and everything built on it (`analysis`, - and `test_materialize.py`'s `engine_dist` wheel build plus the rerun - tests' ephemeral `uv run --with` resolve) does spawn, resolve, and may - touch the network. -- `tests/test_dataset.py` — the storage seam, tested against real tools - via `real_tools`, deliberately: the question is whether bytes land in - the annex or as a blob in git, and a fake answering it would only - restate what the code already believes. Every bug this file found (the - missing `.gitattributes` default, the pointer-file trap, `filter=annex`) - was invisible to a stub. -- `tests/test_project.py` — discovery and convergence semantics, called - directly. This is where scaffold behavior is tested. -- `tests/test_plan.py` tests what lc adds — directories, edges, - `definition_version`, target resolution, the validation gate — and **not** - what a spec means. Scoping, `from:`, `when:` and the recipe grammar are - covered by `astra-tools`' own suite; asserting them here again would - re-create the second implementation this layer just deleted. Every - fixture must be a spec `astra validate` accepts, which the gate now - enforces for free. -- `tests/test_identity.py`, `test_assets.py`, `test_plan.py` — **pure**, - and the whole of layer 2 plus the graph. Nothing spawns, nothing on disk - beyond `tmp_path`. -- `tests/test_worker.py`, `tests/test_materialize.py` — real recipes, - through the real boundary, against a real repository, via the `analysis` - fixture in `conftest.py`. That is the price of testing execution: whether - the gates hold, whether bytes land in the annex, and whether the tree is - clean afterwards are not questions a stub can answer. It is cheap anyway - — the fixture's project declares no dependencies, so `uv lock` and - `uv sync` together cost milliseconds. - - **`cluster_for_run(cluster_id)` is the graph-test monkeypatch point.** - Most tests use an inline scheduler; integration tests borrow real standard - Dask clients, including detached allocation and processes-based cases. -- `tests/test_serve.py` — `lc view` against a **real** detached jupyter - server, its state under `tmp_path`. The page needs no front-end build, - so the lifecycle runs everywhere (the `state` fixture satisfies the - built-page check with an empty file); that the built bundle is served is - gated like the smoke suites (`LC_UI_TESTS_REQUIRED=1` in CI, two guard - tests). The front end itself is checked by driving it in a browser. -- `tests/test_cli.py` — the CLI surface only: flags reaching the engine, - rendering, exit codes, error translation. Rich wraps output at terminal - width, so assert on short unwrappable fragments. -- `tests/test_templates.py` — template loading (guards the packaging of - package data), strict substitution, and the `.gitignore` / - `.gitattributes` entry/repair logic. Content assertions live here; `test_project.py` asserts only that - the file written *is* the template, so a template edit touches one file. - -The sandbox suite splits along the seam, which is what makes it cheap: - -- `tests/test_sandbox_policy.py`, `test_sandbox_wrap.py`, - `test_sandbox_denial.py` — **pure, and run on every OS.** The policy is - data, `wrap` is a function, and the denial renderer is a function, so - the Landlock wrap and the macOS SBPL are both checked on Linux CI with - no privileges and nothing spawned. -- `tests/test_sandbox_shim.py` — the shim as a **real subprocess**, - because its contract *is* its argv and exit codes. Needs no kernel - support: every case here is a setup failure or a pure-function check. -- `tests/test_sandbox_enforcement.py` — **the kernel's answer**, written - once for both mechanisms. See below; this is the one that matters. -- `tests/test_run.py` — what `lc run` decides *before* it execs: the - current-directory project check, declared inputs, the uv hop. Nothing - spawns. -- `tests/test_compute*.py` — resource contracts, malformed native identity, - real detached local lifecycle and fresh-process reconnects, simulated Slurm - commands and a real standard-Dask bootstrap. No Slurm allocation is submitted. - Real command/materialization tests exercise standard task submission and remote - byte forwarding. No site-marker guard or worker-selection layer remains. -- `tests/test_image.py` — **pure**: the declaration, the document, the - render's structure and ordering, tag sensitivity both ways, and the - `env_version` integration. `tests/test_sandbox_oci.py` — **pure**: the - mount table, the argv spellings, the attestation, and the - `contains_prefix` composition through a recorded fake `Popen`. -- `tests/test_container.py` — the image lifecycle against a **stubbed** - `project._run` that models each runtime command's observable effect - (a `save` writes a structurally real docker-archive, a `load` marks - the id present), so every refusal and every strictness is asserted on - recorded argv with nothing spawned. -- `tests/test_container_smoke.py` — **the runtime's answer**, gated like - the enforcement suite: skips without a runtime, - `LC_CONTAINER_TESTS_REQUIRED=1` on Linux CI turns the skip into a hard - failure, and two tests cover the guard itself. Parameterized over the - runtimes present. It builds a real image, commits a real archive into - a real annex, materializes through the real Dask cluster, and runs a - real `datalad rerun` on a bytes-free clone — the record's whole claim. - Each denial sits beside its mutation check. -- `tests/test_crate.py` — **pure**: fixture manifests on `tmp_path`, a - hand-built graph, a stub `writer` — no git anywhere. Structure and - ordering only; the single byte-level assertion is - render-twice-identical, the property convergence rests on. - `tests/test_crate_smoke.py` — **the validator's answer**, gated with - `LC_CRATE_TESTS_REQUIRED=1` (all CI runners — the validator is a dev - dependency, so none may skip): a real materialize, then the official - `rocrate-validator` against the Provenance profile, REQUIRED clean and - RECOMMENDED pinned to the recorded `_FLOOR` set. - -Note the autouse `tools` fixture stubs `engine.project._run` only — -sandbox, allocation, and execution integration tests also spawn real processes -deliberately. Slurm tests simulate native submission commands. - -### The enforcement suite, and why it is shaped that way - -`test_sandbox_enforcement.py` is the only file that can tell you the -layer works. Four properties, each of which it would be easy to lose: - -1. **One suite, both mechanisms.** The same tests run Landlock on Linux - and Seatbelt on macOS, parameterised by `detect()` alone. That is the - seam paying rent — and it is the only way the two stay honest, since - *a leak only Linux catches is a leak*. macOS is in the CI matrix for - exactly this: it is the sole place the generated SBPL is ever - executed. -2. **The real policy.** It runs against `exec_policy` — what an actual - `lc run` *and* an actual recipe get — never a policy hand-built to - make the point. A test - that grants exactly what it is testing cannot discover that the - *shipped* policy grants something else. (This is how `/usr` sat in the - exec set through a full green suite.) -3. **Real leaks, tried literally.** Undeclared *tools* are executed, - undeclared *libraries* are `dlopen`ed, undeclared *data* is read — - the three channels of spec §7. Assertions about path sets belong in - `test_sandbox_policy.py`; this file runs the command. -4. **It cannot pass by not running.** `LC_SANDBOX_TESTS_REQUIRED=1` is - set in CI, which turns "no mechanism here, skip" into a hard failure. - Two tests cover the guard itself, because an unfailing guard is worse - than none. - -**When you add an enforcement test, mutation-check it**: run the same -command through `Unavailable()` and confirm it *succeeds*. A denial test -that would pass unsandboxed is testing nothing, and the failure mode is -silent. Every leak case here was checked that way. - -## Conventions - -- Ruff for linting (E, F, I, N, W, UP), line length 100, target Python 3.11 -- mypy strict mode with `namespace_packages = true`, - `explicit_package_bases = true` -- **Google-style docstrings** on every public function: an imperative - one-line summary, then `Args:` / `Returns:` / `Raises:` / `Yields:` - where they carry information. Concise — a design decision gets a - sentence or two, not its history; the long form belongs in this file. - Two deliberate exceptions: **click command callbacks**, whose docstring - *is* the `--help` text and must stay prose, and properties, whose value - the summary already describes. -- Comments and docstrings carry *why*, never a narrative of how the code - came to be +| A scaffolded file's content | `engine/templates/files/*.tmpl` | `test_templates.py` | +| What gets converged | `engine/project.py` | `test_project.py` | +| How bytes are stored | `engine/dataset.py`, `gitattributes.tmpl` | `test_dataset.py` (real annex) | +| When an output is remade | `engine/assets.py` (ask `astra.api.check`) | `test_assets.py` | +| How the spec becomes a graph | `engine/plan.py` (ask `astra.api.plan`) | `test_plan.py` — lc's additions only, not spec semantics; `test_composition.py` runs 0.1's shapes end to end | +| How a recipe runs / what a run commits | `engine/worker.py` / `engine/materialize.py` | `test_worker.py`, `test_materialize.py` | +| What a sandboxed command may touch | `sandbox/policy.py` | `test_sandbox_policy.py`, `test_sandbox_enforcement.py` | +| A denial message | `sandbox/denial.py` — remedies must work today | `test_sandbox_denial.py` | +| Images | `engine/image.py`, `engine/container.py`, `sandbox/oci.py` | `test_image.py`, `test_container.py`, `test_container_smoke.py` | +| The RO-Crate | `engine/crate.py` (pure, clock-free, sorted) | `test_crate.py`, `test_crate_smoke.py` | +| Where a run executes | `engine/compute/` | `test_compute*.py` | +| What `lc view` shows | `ui/src/index.tsx` | `test_serve.py`, `test_view_browser.py` | +| A CLI verb | `cli/commands.py` — logic stays in the engine | `test_cli.py` (assert short fragments; rich wraps) | diff --git a/README.md b/README.md index cc45a1b3..9e2338c8 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,7 @@ drafted with any AI coding assistant. ## Capabilities - **Multiverse analysis** — declare methodological decisions with multiple defensible options; `lc` materializes your analysis across every universe you define -- **Provenance by construction** — every output is committed to git together with a content-addressed manifest and a re-runnable run record; git-annex carries the bytes, so results travel with the repository +- **Provenance by construction** — every output is committed to git together with its ASTRA provenance record and a re-runnable run record; git-annex carries the bytes, so results travel with the repository - **Locked, isolated execution** — a project's environment is `pyproject.toml` + `uv.lock`; recipes run in it under a sandbox (Landlock on Linux, Seatbelt on macOS) that keeps undeclared files out and stray writes contained - **Containers and HPC** — declare `[tool.lightcone.image]` and recipes run in a content-addressed image archived in the repository itself; allocate local or Slurm Dask clusters explicitly with `lc compute`, then pass the cluster name or full ID to `run` or `materialize` - **Publication view** — declare a license and `lc materialize` maintains an [RO-Crate](https://www.researchobject.org/ro-crate/) of the project and its provenance, ready to archive or deposit diff --git a/pyproject.toml b/pyproject.toml index 55513062..c40b1f00 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -24,7 +24,7 @@ classifiers = [ ] dependencies = [ - "astra-tools==0.2.18", + "astra-tools[evidence]>=0.2.19.dev0", "click>=8.0", "rich>=13.0", "git-annex>=10.2026", @@ -54,9 +54,24 @@ dev = [ ] +# ASTRA 0.1 is not on PyPI yet: both halves are pinned to the commits this +# engine is tested against. astra-tools names astra-spec by a path beside its +# own checkout, which a git source does not have, hence the override. +[tool.uv] +override-dependencies = [ + "astra-spec @ git+https://github.com/LightconeResearch/astra-spec@7de4f21e57a7f23231821a5c3a4b6c560274f36b", +] + +[tool.uv.sources] +astra-tools = { git = "https://github.com/LightconeResearch/astra-tools", rev = "ecbe9504a6ecb8ed303176b51e836d0e4ecc3d9f" } + [project.urls] Repository = "https://github.com/LightconeResearch/lightcone-cli" +# What `astra check` asks about an environment a record names. +[project.entry-points."astra.environments"] +lightcone = "lightcone.engine.identity:SCHEME" + [project.scripts] lc = "lightcone.cli:main" # The git-annex wheel's own entry points, re-declared verbatim: an @@ -105,7 +120,7 @@ explicit_package_bases = true mypy_path = "src" [[tool.mypy.overrides]] -module = ["astra.*", "rocrate.*"] +module = ["rocrate.*"] ignore_missing_imports = true follow_untyped_imports = true diff --git a/src/lightcone/app/__init__.py b/src/lightcone/app/__init__.py index fe1020ec..10fa6b08 100644 --- a/src/lightcone/app/__init__.py +++ b/src/lightcone/app/__init__.py @@ -4,8 +4,8 @@ Jupyter Development Team): a ``LabServerApp`` serving one page, with its token, its page config and Jupyter's Contents API. The page reads the project through that API and resolves the spec itself; the app's own routes -(see ``routes``) answer what that API cannot: an output's recorded run, read -by lc's manifest reader; every output's state, from the engine's own +(see ``routes``) answer what that API cannot: a result's recorded run, read +from its record; every result's state, from the engine's own classification; and the cited papers in astra's cache, outside the project. It serves no settings or themes: the page has no JupyterLab interface to configure. The one other extension it loads is diff --git a/src/lightcone/app/files.py b/src/lightcone/app/files.py index c7c4b3db..c58dc639 100644 --- a/src/lightcone/app/files.py +++ b/src/lightcone/app/files.py @@ -161,7 +161,7 @@ async def get( target = resolved_path(root, root / model["path"]) model["watchPaths"] = [ target.relative_to(root).as_posix(), - assets.manifest_path(Path(model["path"])).as_posix(), + *assets.record_places(model["path"]), ] return model diff --git a/src/lightcone/app/routes.py b/src/lightcone/app/routes.py index 03afa7be..c994254b 100644 --- a/src/lightcone/app/routes.py +++ b/src/lightcone/app/routes.py @@ -1,74 +1,69 @@ """The app's own routes: what the page needs and the Contents API cannot give. -An output's recorded run, read by lc's own manifest reader, so the page -never parses lc's format and the server never has to open dotfiles to it; -every output's state, classified by the engine exactly as ``lc status`` -does, so the page holds no copy of that rule; and the cited papers in -astra's paper cache, which lies outside the -project (``astra paper add`` fills it, and quotes are verified against it). -The paper routes are adapted from jupyterlab-lightcone's ``routes.py`` -(BSD-3-Clause, Lightcone Research), without its fetch route: nothing here -writes the cache. Every route needs the token, like the rest of the server. +A result's recorded run, read from its ASTRA record, so the page never +parses that format and the server never has to open dotfiles to it; +every result's state, classified by the engine exactly as ``lc status`` +does, so the page holds no copy of that rule; and the cited +papers in astra's cache, which lies outside the project (``astra paper +add`` fills it, and quotes are verified against it). Nothing here writes +the cache. Every route needs the token, like the rest of the server. """ from __future__ import annotations import asyncio +import json import os from pathlib import Path from typing import Any -from astra.papers.cache import CachedPaper, PaperCache +from astra.evidence import papers +from astra.records import Record from jupyter_core.utils import ensure_async from jupyter_server.base.handlers import APIHandler from jupyter_server.utils import url_path_join from tornado import web +from tornado.httputil import url_concat -from lightcone.app.files import ProjectFileHandler, open_fd +from lightcone.app.files import open_fd from lightcone.engine import assets, materialize from lightcone.engine.project import ProjectError, current_project -_PAPERS = PaperCache().cache_dir +class PapersHandler(APIHandler): + """Which of the cited DOIs the cache holds a PDF of, and where to get it.""" -def _cached() -> dict[str, CachedPaper]: - """Every cached paper by DOI as the SDK normalizes it, unversioned first.""" - papers: dict[str, CachedPaper] = {} - for paper in PaperCache().list_papers(): - doi = paper.metadata.doi.strip().lower() - if doi not in papers or paper.metadata.version is None: - papers[doi] = paper - return papers + @web.authenticated + async def get(self) -> None: + """Answer the repeated ``doi`` arguments from the cache.""" + found: dict[str, dict[str, str]] = {} + for doi in self.get_arguments("doi"): + if await asyncio.to_thread(papers.get, doi, None) is not None: + url = url_path_join(self.base_url, "lightcone/api/paper") + found[doi] = {"pdfUrl": url_concat(url, {"doi": doi})} + self.set_header("Cache-Control", "private, no-store") + self.finish({"papers": found}) -class PapersHandler(APIHandler): - """Title, authors and PDF of the cited DOIs the cache holds.""" +class PaperHandler(APIHandler): + """The PDF the cache holds for one DOI.""" @web.authenticated async def get(self) -> None: - """Answer the repeated ``doi`` arguments from one scan of the cache.""" - cached = await asyncio.to_thread(_cached) - papers: dict[str, dict[str, str]] = {} - for doi in self.get_arguments("doi"): - if (paper := cached.get(doi)) is None: - continue - pdf = paper.pdf_path.relative_to(_PAPERS).as_posix() - metadata = { - "title": paper.metadata.title, - "authors": ", ".join(paper.metadata.authors or []), - "pdfUrl": url_path_join(self.base_url, "lightcone/papers", pdf), - } - papers[doi] = {key: value for key, value in metadata.items() if value} + """Send the paper the ``doi`` argument names, or a 404.""" + pdf = await asyncio.to_thread(papers.get, self.get_query_argument("doi"), None) + if pdf is None: + raise web.HTTPError(404) self.set_header("Cache-Control", "private, no-store") - self.finish({"papers": papers}) + self.finish(pdf, set_content_type="application/pdf") class RunHandler(APIHandler): - """The run lc recorded for one output, or ``null`` when there is none.""" + """The run recorded for one result, or ``null`` when there is none.""" @web.authenticated async def get(self) -> None: - """Read the manifest beside the output the ``path`` argument names.""" + """Read the record of the result the ``path`` argument names.""" # Whether the path is a file in the project is the contents manager's # to say, as it does for /files/: outside the root, hidden, absent or # a directory is refused before anything is read. @@ -77,25 +72,33 @@ async def get(self) -> None: if not cm.allow_hidden and await ensure_async(cm.is_hidden(path)): raise web.HTTPError(404) model = await ensure_async(cm.get(path, type="file", content=False)) - # The sidecar is hidden, but uses the same confined opener as artifacts. root = Path(cm.root_dir).resolve() - sidecar = assets.manifest_path(root / model["path"]) - try: - with open_fd(root, sidecar, hidden=True) as fd: - with os.fdopen(fd, "r", closefd=False) as file: - manifest = assets.loads(file.read()) - except (web.HTTPError, UnicodeDecodeError): - manifest = None + + def read(name: str) -> str | None: + # The record is hidden, but uses the same confined opener as artifacts. + try: + with open_fd(root, root / name, hidden=True) as fd: + with os.fdopen(fd, "r", closefd=False) as file: + return file.read() + except (web.HTTPError, UnicodeDecodeError): + return None + run: dict[str, Any] | None = None - if manifest is not None: + for record_path in assets.record_places(model["path"]): + if (text := read(record_path)) is None: + continue + try: + record = Record.from_json(json.loads(text)) + except (ValueError, KeyError, TypeError): + continue run = { - "finishedAt": manifest.finished_at, - "gitRevision": manifest.git_sha, - "recipe": manifest.recipe, - "environment": manifest.env_version, - "cliVersion": manifest.lc_version, - "inputVersions": manifest.input_versions, + "finishedAt": record.finished or "", + "gitRevision": record.commit, + "environment": record.environment, + "cliVersion": record.runner.version, + "inputVersions": {item.entry: item.content_hash for item in record.inputs}, } + break self.set_header("Cache-Control", "private, no-store") self.finish({"run": run}) @@ -121,8 +124,7 @@ async def get(self) -> None: HANDLERS = [ ("/lightcone/api/papers", PapersHandler), + ("/lightcone/api/paper", PaperHandler), ("/lightcone/api/run", RunHandler), ("/lightcone/api/status", StatusHandler), - # Jupyter's own file handler: ranges for pdf.js, 304s on reopening. - ("/lightcone/papers/(.*)", ProjectFileHandler, {"path": str(_PAPERS)}), ] diff --git a/src/lightcone/cli/commands.py b/src/lightcone/cli/commands.py index 7e75823e..98485543 100644 --- a/src/lightcone/cli/commands.py +++ b/src/lightcone/cli/commands.py @@ -323,7 +323,7 @@ def materialize( Execution requires a cluster name or ID as its first positional argument. With --check, all positional arguments are targets and no cluster is used. - Each output is committed together with its manifest, in a commit that + Each output is committed together with its record, in a commit that records the command that produced it. The git tree needs to be clean before the run can start. @@ -333,8 +333,8 @@ def materialize( comes out byte-identical stops there instead of cascading. An output made under an earlier environment is reported as behind and - left alone: it is still what the analysis asks for, and the manifest - records the environment and the commit that produced it. Pass + left alone: it is still what the analysis asks for, and its record + names the environment and the commit that produced it. Pass --refresh to remake those too. """ from lightcone.engine import container as engine_container diff --git a/src/lightcone/engine/__init__.py b/src/lightcone/engine/__init__.py index 7e897028..0763a07e 100644 --- a/src/lightcone/engine/__init__.py +++ b/src/lightcone/engine/__init__.py @@ -1,9 +1,3 @@ -"""The lightcone execution engine. - -Rebuild in progress: the engine is being re-added layer by layer on top of -the normative design spec. Layer 1 is project scaffolding only — -:mod:`lightcone.engine.project` (convergence) and -:mod:`lightcone.engine.templates` (the scaffolded files). -""" +"""The lightcone execution engine.""" from __future__ import annotations diff --git a/src/lightcone/engine/assets.py b/src/lightcone/engine/assets.py index ecab1ee3..dc8a5055 100644 --- a/src/lightcone/engine/assets.py +++ b/src/lightcone/engine/assets.py @@ -1,40 +1,34 @@ -"""What a materialized output *is*: where it lives, what it records, and -whether it is still current. - -An asset is a single file — ``results//.`` — -with a manifest sidecar beside it, ``..manifest.json``. The -format comes from the spec, so the path is derived rather than chosen by -the recipe, and one output can only ever be one file. The manifest is the -only part lc writes itself, and it is kept out of the annex so it stays -readable on a clone that has fetched no content at all. - -The rule that classifies an output — ``current``, ``behind`` or -``stale`` — lives here too, next to the manifest it reads and the hashes -it compares. It is the one place in the layer where a bug is quiet rather -than loud: a rule that under-reports leaves an output silently describing -bytes that no longer follow from its inputs. It is a **content-hash** -rule, so a byte-identical rebuild stops the cascade and a restored file -with an old mtime cannot hide. +"""Whether a materialized result is still current. + +A result is a single file at the place ASTRA gives it — +``results//.`` — and its **record**, ASTRA's own +account of how it was made, is at ``.astra/records//.json``. +The record is all lc writes beside a result, and it stays out of the +annex, so it is readable on a clone that has fetched no content at all. + +The rule that classifies a result — ``current``, ``behind`` or ``stale`` — +lives here. Whether a record still describes the analysis is ASTRA's +checker's to say, and it is asked rather than re-derived; lc adds the two +things only an executor knows: the environment a result ran under, and +whether its files were last written by its own run. """ from __future__ import annotations -import hashlib -import json -from collections.abc import Mapping -from dataclasses import asdict, dataclass, field +import posixpath +from collections.abc import Sequence +from dataclasses import dataclass from pathlib import Path -from typing import TYPE_CHECKING, Any, Literal +from typing import TYPE_CHECKING, Literal from lightcone.engine.project import ProjectError if TYPE_CHECKING: + from astra.check import Status as Checked + # A type only: history is git's, and this module runs no git. from lightcone.engine.dataset import LastWrite -MANIFEST_SUFFIX = ".manifest.json" -SCHEMA_VERSION = 1 - #: The marker every annexed path carries, in both of the shapes an annexed #: file takes on disk — the pointer file's first bytes, and the locked #: symlink's target. Detecting the first is the same test git-annex's own @@ -49,166 +43,38 @@ class ContentNotFetchedError(ProjectError): """An annexed file whose content is not in this clone.""" -def output_path(root: Path, universe_id: str, output_id: str, fmt: str) -> Path: - """Locate the file one output is materialized to. +def record_places(path: str) -> list[str]: + """Where the record of a result file at *path* would be. - ``/results//.``. Path-addressed: - this is the path a rendered recipe writes to, with no staging, scratch - or relocation in between. + For a caller holding only a file's path: a result's name is ids + joined by dots and a format may hold dots too, so the file's name + alone does not say where one ends — each split is a candidate, and + the one whose record exists is the answer. Args: - root: The project root. - universe_id: The universe the output is made under. - output_id: The output's id. - fmt: The declared serialization, without a leading dot. + path: A file's POSIX path from the project root. Returns: - The output's path. - - Raises: - ProjectError: If either id cannot name a single path component, or - the format could not be an extension. The path is *composed*, - so an unchecked part would place a file outside the tree the - caller checked — and an id carrying a dot would make - :func:`manifest_path` recover the wrong name. + One record path per split; none for a file that is at no + result's place. """ - for label, value in (("universe", universe_id), ("output", output_id)): - if not value or "/" in value or "\\" in value or value in {".", ".."}: - raise ProjectError( - f"{label} id {value!r} is not a single path component, so it " - f"cannot name a directory under results/." - ) - if "." in output_id: - raise ProjectError( - f"output id {output_id!r} contains a dot, so the manifest beside it " - f"could not be told from the output's own name." - ) - if not fmt or "/" in fmt or "\\" in fmt or fmt.startswith("."): - raise ProjectError( - f"output `{output_id}` declares the format {fmt!r}, which cannot name a " - f"file extension, so the output has nowhere to be written." - ) - return root / "results" / universe_id / f"{output_id}.{fmt}" - - -def manifest_path(output: Path) -> Path: - """The manifest sidecar beside *output*. - - ``..manifest.json``, named from the output's id alone and - never its format — so the manifest keeps its path, and therefore its - history, when a spec re-declares the output in another serialization. - - An id carries no dot (:func:`output_path` refuses one) while a format - may (``tar.gz``), so the id is recovered by partitioning on the - **first** dot. ``Path.stem`` would answer ``x.tar`` for ``x.tar.gz``. - - Args: - output: The output's own path. - - Returns: - The sidecar's path. - """ - return output.parent / f".{output.name.partition('.')[0]}{MANIFEST_SUFFIX}" + *folder, file = path.split("/") + universe = folder[-1:] if folder[-2:-1] == ["results"] else [] + top = folder[: -1 - len(universe)] + if folder[len(top) : len(top) + 1] != ["results"]: + return [] + return [ + "/".join([*top, ".astra", "records", *universe, f"{file[:dot]}.json"]) + for dot, char in enumerate(file) + if char == "." and dot + ] # ============================================================================= -# Content identity +# Content # ============================================================================= -def data_version(path: Path) -> str: - """Hash *path*'s content — its bytes, and nothing else about it. - - A directory hashes each file in sorted relative-path order with the - relative path fed in beside the bytes, so a rename moves the digest. A - file hashes its own bytes, unframed — so the digest is a plain sha256 - an outsider reproduces with ``sha256sum``, and the manifest agrees with - what the crate publishes for the same file. Only the directory side is - framed, which is enough to keep a directory holding one file from - colliding with that file alone. - - Never mtime or size: a content hash is what lets a byte-identical - rebuild stop cascading, and what stops a file restored with an old - timestamp passing as unchanged. - - Args: - path: A file or directory. - - Returns: - The digest, as ``sha256:``. - - Raises: - FileNotFoundError: If *path* does not exist. Never a constant - digest, which would silently disable the staleness chain. - ContentNotFetchedError: If any file is annexed without its content - being in this clone, in either of the shapes that takes. - """ - if path.is_symlink() and not path.exists(): - require_fetched(path) - if not path.exists(): - raise FileNotFoundError(path) - h = hashlib.sha256() - if path.is_file(): - require_fetched(path) - _feed(h, path) - return f"sha256:{h.hexdigest()}" - - h.update(b"dir:") - # A dangling symlink is an unfetched *locked* file, and ``is_file()`` - # answers False for one — so filtering on that alone would drop it from - # the digest without a word, reporting a hash of the subset that happens - # to be present. Only dangling ones are added back: a symlink that - # resolves to a file is already a file, and one that resolves to a - # directory is not content. - files = [ - p - for p in path.rglob("*") - if p.is_file() or (p.is_symlink() and not p.exists()) - ] - for p in sorted(files, key=lambda x: x.relative_to(path).as_posix()): - require_fetched(p) - h.update(b"path:") - h.update(p.relative_to(path).as_posix().encode()) - h.update(b"\0data:") - _feed(h, p) - h.update(b"\0") - return f"sha256:{h.hexdigest()}" - - -class Versions: - """Content identities, computed once per run. - - A declared input is hashed once per ``(universe, output)`` that names - it, which for a multiverse spec is the same bytes over and over: eight - universes times four outputs sharing one catalog reads it thirty-two - times. Memoizing is sound for exactly as long as a run lasts — a run - refuses to start on a dirty tree, and the only in-tree path a recipe - may write is its own output directory, so a declared input's bytes - cannot change underneath it. - - The driver fills the memo before submitting tasks. Its populated dict - travels with each task, so separate worker processes reuse the same hashes. - Failed reads are left to the worker to report for the affected task. - """ - - def __init__(self) -> None: - self._known: dict[Path, str] = {} - - def of(self, path: Path) -> str: - """Return *path*'s content identity, hashing it at most once. - - Args: - path: A declared input, file or directory. - - Returns: - The digest, as ``sha256:``. - """ - resolved = path.resolve() - if (known := self._known.get(resolved)) is None: - known = self._known[resolved] = data_version(path) - return known - - def require_fetched(path: Path) -> None: """Refuse a file whose content this clone does not hold. @@ -264,210 +130,48 @@ def is_pointer(path: Path) -> bool: return f.read(len(_POINTER_PREFIX)) == _POINTER_PREFIX -def _feed(h: hashlib._Hash, path: Path) -> None: - """Stream *path* into *h* — outputs are not assumed to fit in memory.""" - with path.open("rb") as f: - for chunk in iter(lambda: f.read(1 << 16), b""): - h.update(chunk) - - -# ============================================================================= -# The manifest -# ============================================================================= - - -@dataclass(frozen=True) -class Manifest: - """What one materialization recorded about itself. - - Every field is filled by the run that wrote it; there are no optional - halves. ``input_versions`` is the chain — each declared input's content - identity at the moment this output was made — and it is what lets a - change to anything upstream reach here without a timestamp. - """ - - output_id: str - universe_id: str - recipe: str - #: What the spec says this output is: its recipe and decisions. The - #: rebuild trigger. - definition_version: str - #: The environment it was made under. Recorded, never a rebuild - #: trigger — a difference here makes the output *behind*, not stale. - env_version: str - data_version: str - decisions: dict[str, str] - input_versions: dict[str, str] - #: The commit the working tree was at when the run started. Recorded, - #: never hashed into `definition_version` — a commit must not stale - #: every output in the repository, and this is what makes the code that - #: produced a result recoverable anyway. - git_sha: str - #: The `origin` URL, or empty when the repository has no remote. - git_remote: str - #: The engine that made it. Attestation, not identity: outside both - #: hashes, so an lc upgrade neither stales an output nor puts it behind. - lc_version: str - #: What the sandbox actually enforced, as the boundary attested it. - hermeticity: dict[str, Any] - #: When the recipe entered and left the boundary, ISO 8601 UTC with - #: millisecond precision. Attestation, like ``lc_version``: outside - #: both hashes, never a rebuild signal — and defaulted empty because - #: that is the true value for a manifest written before the fields - #: existed, not back-compat machinery. - started_at: str = "" - finished_at: str = "" - #: The uv that resolved and installed the environment the recipe ran - #: in — the one tool between the lock and the installed artifacts. - #: Attestation, like ``lc_version``: outside both hashes, never a - #: rebuild signal, defaulted empty because that is the true value for - #: a manifest written before the field existed. - uv_version: str = "" - #: The image the recipe ran in — ``{tag, id, archive, arch}`` — or - #: ``None`` on the host. Defaulted, and that is not back-compat - #: machinery: ``None`` is the *true* value for every manifest a - #: container-less engine wrote, and without the default those - #: manifests would read as absent and the whole project would go - #: stale over a field that changes nothing about the bytes. - image: dict[str, Any] | None = None - schema_version: int = field(default=SCHEMA_VERSION) - - def as_dict(self) -> dict[str, Any]: - """Return the manifest as JSON-ready data, ``schema_version`` first. - - Returns: - Every field, in declaration order. - """ - data = asdict(self) - return {"schema_version": data.pop("schema_version"), **data} - - -def read(manifest: Path) -> Manifest | None: - """Read the manifest at *manifest*. - - Takes the sidecar's own path rather than the output's, so every caller - holding an output path has to say :func:`manifest_path` out loud. Both - are ``Path``, and a missing file answers ``None`` rather than raising, - so a caller that passed the wrong one would be wrong in silence. - - Args: - manifest: The sidecar's path. - - Returns: - The manifest, or ``None`` when it is absent or unparseable — which - the staleness rule reads as "make it again", the safe direction. - - Raises: - OSError: Deliberately not caught. A permission problem is a real - fault and must not look like an output needing a rebuild. - """ - path = manifest - if not path.is_file(): - return None - try: - return loads(path.read_text()) - except UnicodeDecodeError: - return None - - -def loads(text: str) -> Manifest | None: - """Parse a run record already read by a caller, or return None if invalid.""" - try: - data = json.loads(text) - return Manifest(**{k: v for k, v in data.items() if k in _FIELDS}) - except (json.JSONDecodeError, TypeError, AttributeError, UnicodeDecodeError, ValueError): - # Anything that is not a manifest object reads as "no manifest": - # a top-level array, bytes that are not text, a mapping whose - # fields do not fit. The staleness rule takes that as "make it - # again", which is the safe direction for all of them. - return None - - -def write(path: Path, manifest: Manifest) -> Path: - """Write *manifest* to *path*, atomically. - - The rename is the commit point: a reader sees the previous manifest or - this one, never half of either. The temporary lands beside real outputs - now, so it is named off the sidecar — unique per output, and swept by - the same pathspec. - - Args: - path: The sidecar's path. - manifest: The record to write. - - Returns: - The path written. - """ - temporary = path.with_name(path.name + ".tmp") - temporary.write_text(json.dumps(manifest.as_dict(), indent=2, sort_keys=False) + "\n") - temporary.replace(path) - return path - - -_FIELDS = frozenset(Manifest.__dataclass_fields__) - - # ============================================================================= # Classification — one rule, and it is the only place the rule lives # ============================================================================= # # Three states, and the line between them is what the whole model turns on. # -# An output is **stale** when it contradicts the project as it now stands: -# the spec defines it differently than the artifact was made, or it records -# deriving from bytes the project no longer holds. Either way what is on -# disk is mislabelled, so it is remade. +# A result is **stale** when it contradicts the project as it now stands: +# ASTRA's checker finds no valid record for it, or finds that something it +# is made from has changed since the commit its record names; or its files +# were last committed by something other than its own run. Either way what +# is on disk is mislabelled, so it is remade. # -# An output is **behind** when it is still exactly what the spec asks for, -# but was made under an earlier environment. Nothing about it is wrong — -# the environment it ran under is in its manifest and the commit beside it -# reconstructs that environment — so it is reported and left alone. A -# caller that wants it remade asks for that. +# A result is **behind** when its record still describes the analysis, but +# names an earlier environment. Nothing about it is wrong — the record's +# commit reconstructs that environment — so it is reported and left alone. +# A caller that wants it remade asks for that. # -# The git commit takes part in neither. It is tree-wide, so a README edit -# moves it for every output at once; using it as a signal would mean -# everything is always behind, and a signal that is always on is not one. -# It is recorded, and shown as the context of a `behind` line. +# Whether the bytes are in this clone takes part in neither. A clone that +# has fetched no annex content holds results that are perfectly up to +# date, and a result that was really replaced shows in history. -#: What an output is, relative to the project as it now stands. +#: What a result is, relative to the project as it now stands. Status = Literal["current", "behind", "stale"] -@dataclass(frozen=True) -class Reason: - """Why an output no longer describes what it was made from.""" - - kind: Literal["missing", "definition", "declaration", "input"] - #: Which declared input it was about, for the two input kinds. - input: str = "" - - def __str__(self) -> str: - if self.kind == "missing": - return "no manifest — it has never been materialized" - if self.kind == "definition": - return "the recipe or its decisions changed" - if self.kind == "declaration": - return f"the output no longer declares the same inputs (`{self.input}`)" - return f"the input `{self.input}` changed" - - @dataclass(frozen=True) class Verdict: - """One output's state, and the sentence explaining it.""" + """One result's state, and the sentence explaining it.""" status: Status #: Why, for ``stale`` and ``behind``. Empty for ``current``. why: str = "" def calls_for_a_remake(self, *, refresh: bool) -> bool: - """Whether a run would make this output again. + """Whether a run would make this result again. ``stale`` always, because the artifact contradicts the project. ``behind`` only when asked, because it does not. Args: - refresh: Whether the caller asked for behind outputs too. + refresh: Whether the caller asked for behind results too. Returns: Whether to run the recipe. @@ -475,79 +179,79 @@ def calls_for_a_remake(self, *, refresh: bool) -> bool: return self.status == "stale" or (refresh and self.status == "behind") +def check(root: Path) -> dict[str, tuple[Checked, ...]]: + """Ask ASTRA's checker what every planned result's record says of it. + + Reads records, hashes inputs and reads git; runs nothing and writes + nothing in the project. + + Args: + root: The project root. + + Returns: + Each planned result's statuses, one per place, keyed by the path + of its first place's record from the project root. + + Raises: + ProjectError: If the analysis is missing or breaks one of ASTRA's + rules, or git cannot be read. + """ + from astra import api + + from lightcone.engine import plan + + with plan.asking_astra(root): + report = api.check(root) + if report.errors: + raise ProjectError(plan.invalid(root, report.errors)) + above = posixpath.dirname(report.root) + return { + plan.relative(found.result.places[0].record, above): found.statuses + for found in report.results + } + + def classify( - *, - definition_version: str, - env_version: str, - manifest: Manifest | None, - inputs: Mapping[str, str | None], - foreign: LastWrite | None = None, + *, found: Sequence[Checked], environment: str, foreign: LastWrite | None = None ) -> Verdict: - """Decide what an output is, relative to the project as it now stands. + """Decide what a result is, relative to the project as it now stands. The only place this rule lives. Values rather than objects, because - the callers arrive at them differently and must not diverge in - anything else — history included: whether the output's directory was - last written by its own run record is git's to answer, so it arrives - here as a value computed by whoever has git (the driver), and the - rule stays pure. The *prose* for every verdict lives here too, which - is why this takes the offending commit rather than a finished - sentence. + the driver and the worker arrive at them differently and must not + diverge in anything else — history included: whether the result's + files were last written by its own run is git's to answer, so it + arrives here as a value computed by whoever has git (the driver), and + the rule stays pure. Args: - definition_version: What the spec currently says this output is. - env_version: The run's environment identity. - manifest: The output's recorded manifest, or ``None``. - inputs: Each declared input's current content identity. ``None`` - for an input the caller has already decided will be remade — - check mode's sentinel, meaning "this is going to change", - since it cannot know whether a rebuild is byte-identical. - foreign: The commit that last wrote the output's directory, when - it was not the output's own run record; ``None`` when clean. - A hit is a contradiction — the manifest no longer describes - the bytes — so it is ``stale``, though an output stale on - its definition or inputs keeps that more actionable reason. + found: What ASTRA's checker says at each of the result's places. + environment: The IRI of the project's environment as it now is. + foreign: The commit that last wrote the result's files, when it + was not the result's own run; ``None`` when clean. A hit is a + contradiction — the record no longer describes the bytes — so + it is ``stale``, though a result ASTRA already finds stale + keeps that more actionable reason. Returns: ``stale``, ``behind`` or ``current``, with the reason for the first two. """ - if (reason := _stale(definition_version, manifest, inputs)) is not None: - return Verdict("stale", str(reason)) + if why := [ + reason.message + for status in found + if status.record_status != "unchanged" + for reason in status.reasons + if reason.code != "ABSENT_RESULT" + ]: + return Verdict("stale", "; ".join(dict.fromkeys(why))) if foreign is not None: return Verdict( "stale", f'last changed by {foreign.sha[:7]} ("{foreign.subject}", {foreign.author}, ' - f"{foreign.date}) rather than its run record, so the manifest no longer " + f"{foreign.date}) rather than its own run, so the record no longer " f"describes these bytes — the next run remakes it; inspect first with " f"`git show {foreign.sha[:7]}`", ) - assert manifest is not None # `_stale` returns a reason when it is None - if manifest.env_version != env_version: - # The sentence says what happened; *where* it happened is the - # manifest's `git_sha`, which a caller with a column for it reads - # from the record rather than from prose. + if any(status.record is None or status.record.environment != environment for status in found): return Verdict("behind", "made under an earlier environment") return Verdict("current") - - -def _stale( - definition_version: str, - manifest: Manifest | None, - inputs: Mapping[str, str | None], -) -> Reason | None: - """Why the artifact contradicts the project, or ``None`` if it does not.""" - if manifest is None: - return Reason("missing") - if manifest.definition_version != definition_version: - return Reason("definition") - # The *set* first, and separately: `definition_version` hashes the recipe - # and the decisions, neither of which an input the spec no longer declares - # moves — so without this the loop below would never look at it. Adding an - # input is caught either way; dropping one is only caught here. - if changed := set(inputs) ^ set(manifest.input_versions): - return Reason("declaration", sorted(changed)[0]) - for name, current in inputs.items(): - if current is None or manifest.input_versions[name] != current: - return Reason("input", name) - return None diff --git a/src/lightcone/engine/container.py b/src/lightcone/engine/container.py index b109b482..cd661d40 100644 --- a/src/lightcone/engine/container.py +++ b/src/lightcone/engine/container.py @@ -63,8 +63,6 @@ class Runtime: #: The image id (bare hex of its config blob) — execution pins on #: this, never the tag, so a retagged image cannot substitute. image_id: str = "" - #: The architecture the archive was built for. - arch: str = "" @property def supports_gpus(self) -> bool: @@ -80,19 +78,6 @@ def archive(self) -> str: so this cannot become a second spelling of the layout.""" return image.archive_path(self.root, self.image_tag).relative_to(self.root).as_posix() - def manifest_image(self) -> dict[str, str] | None: - """This world as the manifest's ``image`` field; ``None`` on the - host. Beside the data, so a field added here cannot be forgotten - at the write site — the ``asdict(attestation)`` discipline.""" - if self.mode == "direct": - return None - return { - "tag": self.image_tag, - "id": self.image_id, - "archive": self.archive, - "arch": self.arch, - } - def runtime_for_run(root: Path, *, build: bool, use_gpus: bool = False) -> Runtime: """Resolve the execution world, converging the image where allowed. @@ -156,7 +141,6 @@ def runtime_for_run(root: Path, *, build: bool, use_gpus: bool = False) -> Runti runtime=name, image_tag=tag, image_id=image_id, - arch=arch, ) diff --git a/src/lightcone/engine/crate.py b/src/lightcone/engine/crate.py index 473b490c..52b50363 100644 --- a/src/lightcone/engine/crate.py +++ b/src/lightcone/engine/crate.py @@ -1,15 +1,15 @@ """The publication view: the repository described as a Workflow Run RO-Crate. The project *is* the crate. ``ro-crate-metadata.json`` sits at the root -and describes what the repository already holds — the spec, the lock, the -universes, each materialized output and the run that made it — so a -deposit is ``git archive`` on the repository, not an export step that -copies bytes. lc's manifests stay the canonical record; the crate is the -same facts in schema.org vocabulary, readable by archives and viewers -that will never run ``lc``. +and describes what the repository already holds — the spec, the lock, +each materialized result and the run that made it — so a deposit is +``git archive`` on the repository, not an export step that copies bytes. +ASTRA's records stay the canonical account; the crate is the same facts +in schema.org vocabulary, readable by archives and viewers that will +never run ``lc``. The document is a pure function of repository state: :func:`render` -takes the graph, the license, and a way to ask git who last wrote each +takes the graph, the license, the remote, and a way to ask git who last wrote each output, and returns bytes that are identical when nothing changed. That is what lets ``lc materialize`` converge the file the way it converges ``uv.lock`` — compare, and commit only a difference — with no verb for a @@ -17,9 +17,9 @@ Profiles targeted: Process, Workflow and Provenance Run Crate 0.5, plus Workflow RO-Crate 1.0. The Provenance layer is structural, not claimed: -one ``lc materialize`` invocation is one ``OrganizeAction`` (outputs of a -run share their manifests' ``git_sha``, because the driver reads HEAD -once and hands it down), each output's execution is a ``CreateAction`` +one ``lc materialize`` invocation is one ``OrganizeAction`` (results of a +run share their records' ``commit``, because the driver reads HEAD +once and hands it down), each result's execution is a ``CreateAction`` steered by a ``ControlAction``, and the spec's outputs are the workflow's ``HowToStep`` s. """ @@ -28,21 +28,25 @@ import hashlib import json +import posixpath import re import uuid from collections.abc import Callable, Mapping from pathlib import Path -from typing import Any +from typing import TYPE_CHECKING, Any from rocrate.model import ContextEntity from rocrate.model.computerlanguage import ComputerLanguage from rocrate.rocrate import ROCrate -from lightcone.engine import assets, plan +from lightcone.engine import assets from lightcone.engine.dataset import LastWrite from lightcone.engine.plan import Graph, Key from lightcone.engine.project import SPEC_FILENAME +if TYPE_CHECKING: + from astra.records import Record + #: The vocabulary the run-level facts come from. Without it in the #: ``@context``, terms like ``containerImage`` and ``sha256`` are #: undefined and silently dropped on JSON-LD expansion — typed but inert. @@ -86,6 +90,7 @@ def render( *, license: str, dsid: str, + remote: str, writer: Callable[..., LastWrite], keys: Mapping[str, str], ) -> str: @@ -93,15 +98,17 @@ def render( Pure given its arguments: reads the tree, runs nothing, and returns identical bytes for identical repository state — timestamps come from - manifests and commits, never from the clock, and entities are built + records and commits, never from the clock, and entities are built in sorted order. The writer is asked about each path at most once. Args: root: The project root. - graph: The full task graph — every universe, every output. + graph: The full task graph — every planned result. license: The declared license, from :func:`project.license_of`. dsid: The dataset UUID, the namespace absolute entity ids are minted under so they are stable across clones. + remote: The repository's ``origin`` URL, or empty when it has + none — where the workflow and its recipes can be found. writer: Answers "which commit last touched this path" — :func:`dataset.last_writer` bound to the root, injected so the builder stays free of git. @@ -113,12 +120,33 @@ def render( Returns: The ``ro-crate-metadata.json`` text, trailing newline included. """ - build = _Builder(root, graph, license, dsid, writer, keys) + build = _Builder(root, graph, license, dsid, remote, writer, keys) return build.document() -def _control_id(key: Key) -> str: - return f"#control-{key[0]}-{key[1]}" +def _split(key: Key) -> tuple[str, str]: + """A task's result name — after its analysis's folder, for a result + that belongs to another analysis — and its universe, empty for a + result built once.""" + name, _, universe = key.partition("#") + return name, universe + + +def _slug(key: Key) -> str: + """A task's key as a fragment identifier, which cannot hold its ``#``.""" + name, universe = _split(key) + return f"{universe}-{name}" if universe else name + + +def _offset(stamp: str) -> str: + """A record's UTC instant with a numeric offset: the Run Crate shapes + read ``startTime`` and ``endTime`` with ``+hh:mm``, never ``Z``.""" + return stamp.removesuffix("Z") + "+00:00" + + +def _titled(key: Key) -> str: + name, universe = _split(key) + return f"`{name}` in universe `{universe}`" if universe else f"`{name}`" class _Builder: @@ -130,10 +158,15 @@ def __init__( graph: Graph, license: str, dsid: str, + remote: str, writer: Callable[..., LastWrite], keys: Mapping[str, str], ) -> None: - from astra.helpers import load_yaml + from astra.contract import Contract + from astra.diagnostics import Catalogue + from astra.load import ParseError, load_yaml + from astra.records import Record, read_record + from astra.sources.reader import WorktreeReader self.root = root self.graph = graph @@ -142,30 +175,29 @@ def __init__( self.keys = dict(keys) self.crate = ROCrate(version=_RO_CRATE_VERSION) self.crate.metadata.extra_contexts.append(_WORKFLOW_RUN_CONTEXT) - #: Every materialized task, sorted: the one iteration order. - self.made: list[tuple[Key, assets.Manifest]] = sorted( - ( - (key, manifest) - for key, task in graph.tasks.items() - if (manifest := assets.read(task.manifest_path)) is not None - ), - key=lambda pair: pair[0], - ) + #: Every materialized task, sorted, with its record at the first + #: of its places — the one its recipe wrote; the others hold copies. + reader, contract = WorktreeReader(root, None), Contract.load(Catalogue.load()) + self.made: list[tuple[Key, Record]] = [] + for key in sorted(graph.tasks): + record = read_record(reader, graph.tasks[key].places[0].record, contract) + if isinstance(record, Record): + self.made.append((key, record)) self.made_keys = {key for key, _ in self.made} self.persons: dict[str, str] = {} # email/name → @id self.agents: dict[Key, str] = {} # each action's Person @id, or "" - self.images: set[str] = set() # archive paths already added self.engines: set[str] = set() # engine @ids already added self.actions: list[str] = [] # action @ids, for the root's mentions #: Loop invariants, asked once: the uuid5 namespace every minted - #: id lives under, the project's one recorded remote, the spec's - #: header and last commit, and which decisions it declares. + #: id lives under, the spec's header and last commit, and which + #: decisions it declares. self.namespace = uuid.uuid5(uuid.NAMESPACE_URL, dsid) - remotes = {m.git_remote for _, m in self.made if m.git_remote} - self.remote = remotes.pop() if len(remotes) == 1 else "" - loaded = load_yaml(root / SPEC_FILENAME) - self.spec: dict[str, Any] = dict(loaded) if isinstance(loaded, dict) else {} - self.title = str(self.spec.get("name") or root.name) + self.remote = remote + try: + self.spec: dict[str, Any] = load_yaml((root / SPEC_FILENAME).read_bytes()).value + except ParseError: + self.spec = {} + self.title = str(self.spec.get("label") or root.name) self.spec_write = writer(root / SPEC_FILENAME) self.parameters = {d for t in graph.tasks.values() for d in t.decisions} @@ -174,10 +206,10 @@ def document(self) -> str: workflow = self._workflow() self._steps_and_tools(workflow) self._environment_files() - for key, manifest in self.made: - self._dataset(key, manifest) - for key, manifest in self.made: - self._control(key, self._action(key, manifest)) + for key, record in self.made: + self._dataset(key, record) + for key, record in self.made: + self._control(key, self._action(key, record)) self._runs(workflow) self._root() document = self.crate.metadata.generate() @@ -253,7 +285,7 @@ def _workflow(self) -> Any: return workflow def _steps_and_tools(self, workflow: Any) -> None: - output_ids = sorted({output_id for _, output_id in self.graph.tasks}) + output_ids = sorted({_split(key)[0] for key in self.graph.tasks}) for position, output_id in enumerate(output_ids): tool = ContextEntity( self.crate, @@ -295,12 +327,16 @@ def _tool_id(self, output_id: str) -> str: # ----- the data entities ----- def _environment_files(self) -> None: - """The lock and its companions — what every run consumed.""" - universes = sorted( - path.relative_to(self.root).as_posix() - for path in (self.root / "universes").glob("*.yaml") + """The lock and its companions — what every run consumed — and + the spec of each analysis below the project's whose recipes run.""" + specs = sorted( + { + posixpath.join(task.workdir, SPEC_FILENAME) + for task in self.graph.tasks.values() + if task.workdir + } ) - for name in (*_ENVIRONMENT, *universes): + for name in (*_ENVIRONMENT, *specs): if (self.root / name).is_file(): self._file(name) if (self.root / "README.md").is_file(): @@ -350,61 +386,54 @@ def _integrity(self, name: str) -> dict[str, str]: return {"contentSize": str(len(data)), "sha256": hashlib.sha256(data).hexdigest()} def _dataset_id(self, key: Key) -> str: - """One output's crate id — :func:`plan.declared_path`'s answer, - never a second spelling of the results layout. No trailing slash: - an output is one file, so the entity is a ``File``.""" - return plan.declared_path(self.root, self.graph.tasks[key].output_path) - - def _dataset(self, key: Key, manifest: assets.Manifest) -> None: - universe_id, output_id = key - task = self.graph.tasks[key] - # A `File`, not a `Dataset` with parts: an output is one file, so + """One result's crate id — the place ASTRA gives its bytes, never + a second spelling of the results layout. No trailing slash: a + result is one file, so the entity is a ``File``.""" + return self.graph.tasks[key].places[0].bytes + + def _dataset(self, key: Key, record: Record) -> None: + place = self.graph.tasks[key].places[0] + # A `File`, not a `Dataset` with parts: a result is one file, so # there is one annex key to look up by name and one checksum to - # publish. `data_version` is that same digest — the manifest and - # the crate no longer carry two numbers for the same bytes. + # publish — the same digest its record gives. entity = self._file(self._dataset_id(key)) - entity["name"] = f"{output_id} (universe {universe_id})" - entity["description"] = f"output `{output_id}` materialized under `{universe_id}`" - entity["version"] = manifest.data_version - manifest_file = self._file(plan.declared_path(self.root, task.manifest_path)) - manifest_file["about"] = {"@id": entity.id} - entity["subjectOf"] = {"@id": manifest_file.id} + entity["name"] = record.address + entity["description"] = f"output {_titled(key)}, materialized" + entity["version"] = record.content_hash + described = self._file(place.record) + described["about"] = {"@id": entity.id} + entity["subjectOf"] = {"@id": described.id} # ----- the runs ----- - def _action(self, key: Key, manifest: assets.Manifest) -> str: - universe_id, output_id = key - task = self.graph.tasks[key] - write = self.writer(task.output_path, task.manifest_path) + def _action(self, key: Key, record: Record) -> str: + place = self.graph.tasks[key].places[0] + write = self.writer(place.bytes, place.record) self.agents[key] = self._person(write.author, write.email) if write else "" properties: dict[str, Any] = { "@type": "CreateAction", - "name": f"run of `{output_id}` in universe `{universe_id}`", - # The *recorded* recipe, not the graph's: the action states - # what ran, and the spec may have moved since — the manifest - # is the canonical record of the execution. - "description": manifest.recipe, - "instrument": {"@id": self._tool_id(output_id)}, + "name": f"run of {_titled(key)}", + # The command as the spec now stands, like the decisions below: + # a record holds neither, and one made from another command is + # stale, so this is what a current result ran. + "description": self.graph.tasks[key].command, + "instrument": {"@id": self._tool_id(_split(key)[0])}, "result": [{"@id": self._dataset_id(key)}], "actionStatus": "http://schema.org/CompletedActionStatus", - "object": self._objects(key, manifest), + "object": self._objects(key), } - if manifest.started_at: - properties["startTime"] = manifest.started_at - if manifest.finished_at: - properties["endTime"] = manifest.finished_at + if record.started: + properties["startTime"] = _offset(record.started) + if record.finished: + properties["endTime"] = _offset(record.finished) if self.agents[key]: properties["agent"] = {"@id": self.agents[key]} - if manifest.image is not None: - properties["containerImage"] = {"@id": self._image(manifest.image)} - action = ContextEntity( - self.crate, self._uri(f"action/{universe_id}/{output_id}"), properties - ) + action = ContextEntity(self.crate, self._uri(f"action/{key}"), properties) self.crate.add(action) self.actions.append(str(action.id)) return str(action.id) - def _objects(self, key: Key, manifest: assets.Manifest) -> list[dict[str, str]]: + def _objects(self, key: Key) -> list[dict[str, str]]: task = self.graph.tasks[key] refs = [{"@id": name} for name in _ENVIRONMENT if (self.root / name).is_file()] for name in sorted(task.inputs): @@ -413,85 +442,61 @@ def _objects(self, key: Key, manifest: assets.Manifest) -> list[dict[str, str]]: if upstream in self.made_keys: refs.append({"@id": self._dataset_id(upstream)}) continue - refs.append({"@id": self._external(name, task.inputs[name])}) - # The *recorded* decisions: the values the recipe actually ran - # under, whatever the spec says today. A decision the workflow no - # longer declares gets no exampleOfWork — there is no parameter - # for it to exemplify. - for decision in sorted(manifest.decisions): + refs.append({"@id": self._input(task.inputs[name])}) + # The options the task's decisions select as the spec now stands: + # a record holds none, and one that names another selection is + # stale, so these are what a current result ran under. + for decision in sorted(task.decisions): properties: dict[str, Any] = { "@type": "PropertyValue", "name": decision, - "value": manifest.decisions[decision], + "value": task.decisions[decision], + "exampleOfWork": {"@id": f"#param-{decision}"}, } - if decision in self.parameters: - properties["exampleOfWork"] = {"@id": f"#param-{decision}"} - value = ContextEntity( - self.crate, f"#value-{key[0]}-{key[1]}-{decision}", properties - ) + value = ContextEntity(self.crate, f"#value-{_slug(key)}-{decision}", properties) self.crate.add(value) refs.append({"@id": value.id}) return refs - def _external(self, name: str, path: Path) -> str: - """A declared input the spec points at, in or out of the tree. - - In-or-out is :func:`plan.declared_path`'s answer — relative - inside the tree, absolute outside it — never a second spelling - of that rule here: two copies of one path rule is how the first - one shipped a bug. - - An in-tree input's checksum comes from its annex key, like every - other file: a ``File`` entity's ``sha256`` describes the - *deposit* — the bytes a ``git archive`` carries — never what any - particular run consumed, so manifests disagreeing about a shared - input (a half-rebuilt project) do not suppress it; which bytes a - run consumed is its own manifest's ``input_versions``. An - out-of-tree input carries none: its recorded digest is lc's - *framed* hash, not a raw sha256, so publishing it under the - workflow-run ``sha256`` term would be a checksum nothing can - verify — the manifests keep the full story, which is the layer's - stated weaker promise. + def _input(self, path: str) -> str: + """A declared input file, by its path in the repository. + + Its checksum comes from its annex key, like every other file: a + ``File`` entity's ``sha256`` describes the *deposit* — the bytes a + ``git archive`` carries — never what any particular run consumed, + so records disagreeing about a shared input (a half-rebuilt + project) do not suppress it; which bytes a run consumed is its + own record's ``inputs``. """ - declared = plan.declared_path(self.root, path) - in_tree = not Path(declared).is_absolute() - entity_id = declared if in_tree else Path(declared).as_uri() - if self.crate.dereference(entity_id) is not None: - return entity_id - properties: dict[str, Any] = {"@type": "File", "name": declared} - if fmt := _format_of(declared): - properties["encodingFormat"] = fmt - if in_tree: - properties.update(self._integrity(declared)) - self.crate.add_file(path, declared, properties=properties) - else: - # Outside the repository: recorded by content, not stored - # in it — the layer's stated weaker promise, so a context - # entity rather than a data entity the crate cannot hold. - self.crate.add(ContextEntity(self.crate, entity_id, properties)) - return entity_id + if self.crate.dereference(path) is None: + properties: dict[str, Any] = {"@type": "File", "name": path} + if fmt := _format_of(path): + properties["encodingFormat"] = fmt + properties.update(self._integrity(path)) + self.crate.add_file(self.root / path, path, properties=properties) + return path def _control(self, key: Key, action_id: str) -> None: - universe_id, output_id = key self.crate.add( ContextEntity( self.crate, - _control_id(key), + f"#control-{_slug(key)}", properties={ "@type": "ControlAction", - "name": f"orchestration of `{output_id}` in universe `{universe_id}`", - "instrument": {"@id": f"#step-{output_id}"}, + "name": f"orchestration of {_titled(key)}", + "instrument": {"@id": f"#step-{_split(key)[0]}"}, "object": {"@id": action_id}, }, ) ) def _runs(self, workflow: Any) -> None: - """One ``OrganizeAction`` per run — outputs sharing a ``git_sha`` - were made by one ``lc materialize``, the driver's one HEAD read.""" - runs: dict[str, list[tuple[Key, assets.Manifest]]] = {} - for key, manifest in self.made: - runs.setdefault(manifest.git_sha, []).append((key, manifest)) + """One ``OrganizeAction`` per run — results whose records name one + commit were made by one ``lc materialize``, the driver's one HEAD + read.""" + runs: dict[str, list[tuple[Key, Record]]] = {} + for key, record in self.made: + runs.setdefault(record.commit, []).append((key, record)) for sha in sorted(runs): group = runs[sha] # The engine *that run* recorded, never the one installed @@ -499,45 +504,43 @@ def _runs(self, workflow: Any) -> None: # or two collaborators on different lc versions re-commit # the crate at each other forever — and an lc upgrade is # recorded as moving nothing. One run has one engine by - # construction; every manifest of the group agrees. - engine = self._engine(group[0][1].lc_version) - times = sorted(t for _, m in group for t in (m.started_at, m.finished_at) if t) + # construction; every record of the group agrees. + engine = self._engine(group[0][1].runner.version) + times = sorted(t for _, r in group for t in (r.started, r.finished) if t) run_action = ContextEntity( self.crate, self._uri(f"run/{sha}"), properties={ "@type": "CreateAction", - "name": f"lc materialize at {sha[:7] or 'unknown commit'}", - "description": ( - f"one materialization run, starting from commit {sha or '(unrecorded)'}" - ), + "name": f"lc materialize at {sha[:7]}", + "description": f"one materialization run, starting from commit {sha}", "instrument": {"@id": workflow.id}, "result": [{"@id": self._dataset_id(key)} for key, _ in group], "actionStatus": "http://schema.org/CompletedActionStatus", }, ) if times: - run_action["startTime"] = times[0] - run_action["endTime"] = times[-1] + run_action["startTime"] = _offset(times[0]) + run_action["endTime"] = _offset(times[-1]) if agent := self.agents.get(group[0][0], ""): run_action["agent"] = {"@id": agent} self.crate.add(run_action) self.actions.append(str(run_action.id)) organize = ContextEntity( self.crate, - f"#organize-{sha[:7] or 'unknown'}", + f"#organize-{sha[:7]}", properties={ "@type": "OrganizeAction", - "name": f"scheduling of the run at {sha[:7] or 'unknown commit'}", + "name": f"scheduling of the run at {sha[:7]}", "instrument": {"@id": engine}, - "object": [{"@id": _control_id(key)} for key, _ in group], + "object": [{"@id": f"#control-{_slug(key)}"} for key, _ in group], "result": {"@id": run_action.id}, }, ) self.crate.add(organize) def _engine(self, version: str) -> str: - """The engine one run's manifests attest, added once per version. + """The engine one run's records name, added once per version. Its release page is both the id and the ``url`` — the one address derivable from repository state alone; anything read off the @@ -560,25 +563,6 @@ def _engine(self, version: str) -> str: self.engines.add(engine_id) return engine_id - def _image(self, image: dict[str, Any]) -> str: - """The committed archive: identity and payload as one entity.""" - archive = str(image.get("archive") or "") - if archive not in self.images: - properties: dict[str, Any] = { - "@type": ["File", "ContainerImage"], - "name": str(image.get("tag") or archive), - "tag": str(image.get("tag") or ""), - "encodingFormat": "application/x-tar", - # The archive is `docker-archive` format wherever it runs — - # podman, docker and podman-hpc all consume it as one. - "additionalType": {"@id": "https://w3id.org/ro/terms/workflow-run#DockerImage"}, - } - if str(image.get("id") or "").startswith("sha256:"): - properties["sha256"] = str(image["id"]).removeprefix("sha256:") - self.crate.add_file(self.root / archive, archive, properties=properties) - self.images.add(archive) - return archive - # ----- shared entities and the root ----- def _person(self, author: str, email: str) -> str: @@ -606,7 +590,7 @@ def _root(self) -> None: # The newest recorded instant, never the clock: the document must # be a pure function of repository state, or every render is a # fresh diff and convergence commits forever. - stamps = sorted(m.finished_at for _, m in self.made if m.finished_at) + stamps = sorted(r.finished for _, r in self.made if r.finished) root["datePublished"] = stamps[-1] if stamps else (self.spec_write.date or "1970-01-01") root["conformsTo"] = [{"@id": profile_id} for profile_id, _, _ in _PROFILES] for profile_id, name, version in _PROFILES: diff --git a/src/lightcone/engine/dataset.py b/src/lightcone/engine/dataset.py index be638329..399c02c7 100644 --- a/src/lightcone/engine/dataset.py +++ b/src/lightcone/engine/dataset.py @@ -4,7 +4,7 @@ history, git-annex carries the bytes — reached through ordinary ``git`` and ``git annex`` commands. ``.gitattributes`` routes content: the default is ``annex.largefiles=nothing``, and outputs and declared inputs opt out -of it, so manifests and analysis code stay in git while results and data +of it, so records and analysis code stay in git while results and data go to the annex. Every command goes through :func:`~lightcone.engine.project._run`, the @@ -230,8 +230,7 @@ def head(directory: Path) -> tuple[str, str]: Returns: ``(commit sha, origin URL)``. The URL is empty rather than absent - when the repository has no remote — a manifest field that is - sometimes missing reads worse than one that is sometimes blank. + when the repository has no remote. """ remote = project._run(["git", "config", "--get", "remote.origin.url"], cwd=directory) return ( @@ -273,10 +272,10 @@ def last_writer(directory: Path, *paths: Path | str) -> LastWrite: ``.git``, or a host without git — states such projects are actually in. - Several paths, because an output is a file *and* a manifest and a - hand edit to either contradicts the record: the payload alone would - miss a doctored manifest, and the manifest alone would miss the forged - result that is the whole reason to ask. The manifest also carries the + Several paths, because a result is a file *and* its record and a + hand edit to either contradicts the other: the payload alone would + miss a doctored record, and the record alone would miss the forged + result that is the whole reason to ask. The record also carries the history across a re-declared format, where the payload's path is new. Args: @@ -331,29 +330,21 @@ def annex_keys(directory: Path) -> dict[str, str]: return keys -def tracked_manifests(directory: Path) -> list[Path]: - """Every manifest sidecar git is tracking, anywhere in the tree. +def tracked_records(directory: Path) -> list[str]: + """Every record git is tracking under the project's own ``.astra/``. - From the index rather than a directory walk: it is exact, it costs one - process, and it never descends into ``.git`` or ``.venv``. Scoped to - ``results/``, which is the only place lc writes one — elsewhere the - name is the user's to use. Untracked strays are the dirty check's - business, not this one's. + From the index rather than a directory walk: it is exact and it costs + one process. Untracked strays are the dirty check's business, not + this one's. Args: - directory: The repository root. + directory: The project root. Returns: - Absolute paths, unordered. + Paths from the project root, unordered. """ - # Imported here, not at module scope: `assets` reaches this module - # through `project`, so the name has to be fetched after both exist. - from lightcone.engine.assets import MANIFEST_SUFFIX - - listed = _git( - ["ls-files", "-z", "--", f":(glob)results/**/*{MANIFEST_SUFFIX}"], cwd=directory - ) - return [directory / name for name in listed.split("\0") if name] + listed = _git(["ls-files", "-z", "--", ":(glob).astra/records/**/*.json"], cwd=directory) + return [name for name in listed.split("\0") if name] def save(directory: Path, paths: Iterable[Path | str], message: str) -> bool: @@ -380,8 +371,7 @@ def save(directory: Path, paths: Iterable[Path | str], message: str) -> bool: into its output directory, would land as a full blob in git, silently, with every test green, and every clone would carry the bytes forever. With it, ``annex.largefiles`` alone decides — which is - the storage policy the ``.gitattributes`` template already states, - dot-named manifests staying in git through their own exemption. + the storage policy the ``.gitattributes`` template already states. Per-add and never written to the repository's config, so a user's own ``git add`` keeps git-annex's stock behavior. diff --git a/src/lightcone/engine/identity.py b/src/lightcone/engine/identity.py index 55cd39d4..59f5239c 100644 --- a/src/lightcone/engine/identity.py +++ b/src/lightcone/engine/identity.py @@ -1,11 +1,4 @@ -"""What a materialized output is identified by. - -Two hashes, and they answer different questions on purpose. - -``definition_version`` is what the spec says an output *is*: its rendered -recipe and the decisions it was made under. It is the rebuild trigger — -when it moves, the artifact on disk is no longer an instance of the thing -the spec describes, so keeping it would be mislabelling it. +"""The environment a result ran under, and how far its identity reaches. ``env_version`` is the environment's identity: the lock's bytes, the interpreter pin's bytes, the settings that decide *which artifacts* @@ -15,25 +8,24 @@ the alternative is a parse that silently disagrees with uv about what the lock means. -**``env_version`` is not part of ``definition_version``**, and that is the -whole shape of the model. An environment moves for reasons that have -nothing to do with a given output — one added dependency rewrites the lock -for the whole project — and a research artifact costs hours to remake and -is often already looked at. So an environment edit does not stale -anything; it makes an artifact *behind*, which is a fact the report states -and a rebuild the caller can ask for. Over-sensitivity is affordable -exactly because it no longer spends compute. - -What makes that safe is that nothing is lost: the environment an output -was made under is recorded in its manifest, and the commit alongside it -reconstructs that environment from the lock of the day. - -The git commit is not an input to either hash. It is tree-wide, so -hashing it would stale every output in the repository on a README edit. - -Both hashes are length-framed. Concatenating fields raw lets a boundary -shift between them produce the same digest from different inputs — a -recipe ending in a character the decisions begin with, say — and nothing +A record names the environment its recipe ran in by an IRI, and +:func:`environment` is that IRI: ``env_version`` under lc's own ``urn:`` +prefix. ASTRA records the environment and never compares it, **and +neither does the rebuild rule** — that is the whole shape of the model. +An environment moves for reasons that have nothing to do with a given +output — one added dependency rewrites the lock for the whole project — +and a research artifact costs hours to remake and is often already looked +at. So an environment edit does not stale anything; it makes an artifact +*behind*, which is a fact the report states and a rebuild the caller can +ask for. Over-sensitivity is affordable exactly because it no longer +spends compute. + +What makes that safe is that nothing is lost: every file the hash covers +is committed, so the commit a record names reconstructs the environment +its IRI names from the lock of the day. + +The hash is length-framed. Concatenating fields raw lets a boundary shift +between them produce the same digest from different inputs, and nothing about a content hash is worth having if it can be shifted. """ @@ -44,7 +36,6 @@ import os import re import tomllib -from collections.abc import Mapping from dataclasses import dataclass from pathlib import Path from typing import Any @@ -109,32 +100,51 @@ def env_version(root: Path) -> str: return f"sha256:{h.hexdigest()}" -def definition_version(*, recipe: str, decisions: Mapping[str, str], fmt: str) -> str: - """Compute what the spec says one output is. +_IRI = "urn:lightcone:env:" - ``sha256(recipe ‖ canonical decisions ‖ format)``, length-framed. The - environment is deliberately absent: see this module's docstring. - The format is framed in its own right rather than left to arrive - through the rendered recipe's ``{output}`` path, because a recipe need - not use that placeholder at all. One that does not would keep its - digest across a re-declared serialization while the manifest — whose - sidecar is named from the id alone, so it does not move either — went - on describing an output at a path that no longer exists. +def environment(root: Path) -> str: + """Name the project's environment as a record does. Args: - recipe: The rendered recipe command. - decisions: The decisions this output declares, as id → option. - fmt: The declared serialization. + root: The project root. Returns: - The digest, as ``sha256:``. + The IRI: :func:`env_version` under lc's ``urn:`` prefix. + + Raises: + ProjectError: If ``uv.lock`` or ``.python-version`` is missing. """ - h = hashlib.sha256() - _frame(h, "recipe", recipe.encode()) - _frame(h, "decisions", _canonical(dict(decisions)).encode()) - _frame(h, "format", fmt.encode()) - return f"sha256:{h.hexdigest()}" + return _IRI + env_version(root) + + +class Scheme: + """What lc answers ASTRA's checker about an environment it recorded. + + :data:`SCHEME` is registered in the ``astra.environments`` entry-point + group, which is how ``astra check`` learns that an IRI is one a runner + can recreate. + """ + + name = "lightcone" + + def recognises(self, iri: str) -> bool: + """Whether *iri* is one lc records.""" + return iri.startswith(_IRI) + + def holds(self, iri: str) -> str: + """Say what an environment lc recorded is made of.""" + return ( + "the uv environment of the record's commit: uv.lock, .python-version, uv's " + "install settings and, in a containerized project, the image definition" + ) + + def recreatable(self, iri: str) -> bool: + """Every file the hash covers is in the record's commit.""" + return True + + +SCHEME = Scheme() def _frame(h: hashlib._Hash, label: str, data: bytes) -> None: diff --git a/src/lightcone/engine/materialize.py b/src/lightcone/engine/materialize.py index 60247ca1..21606a96 100644 --- a/src/lightcone/engine/materialize.py +++ b/src/lightcone/engine/materialize.py @@ -33,7 +33,9 @@ import functools import json +import posixpath import re +import shlex from collections.abc import Iterable, Iterator, Sequence from contextlib import contextmanager from dataclasses import asdict, dataclass, field, replace @@ -47,6 +49,8 @@ from lightcone.engine.project import ProjectError if TYPE_CHECKING: + from astra.records import Record + from lightcone.engine.compute.output import Forwarder @@ -54,20 +58,20 @@ class MaterializeReport: """What a run did, or — in check mode — what it would do.""" - #: ``universe/output`` for each output this run produced and committed. + #: Each result this run produced and committed, as lc names its task. made: list[str] = field(default_factory=list) #: Nothing about them changed, and the environment is the one they were #: made under. current: list[str] = field(default_factory=list) #: Still what the spec asks for, but made under an earlier environment - #: — ``universe/output`` → the context, naming the commit. Left alone; + #: — task → the context. Left alone; #: ``refresh`` is what remakes them. behind: dict[str, str] = field(default_factory=dict) #: The recipe failed, or the environment moved under it. failed: list[str] = field(default_factory=list) #: Not attempted, because something upstream did not finish. blocked: list[str] = field(default_factory=list) - #: Check mode only: ``universe/output`` → why it would run. + #: Check mode only: task → why it would run. planned: dict[str, str] = field(default_factory=dict) #: lc's own prose: what the lock scan found, why a task did not finish. warnings: list[str] = field(default_factory=list) @@ -117,165 +121,106 @@ def check(root: Path, targets: Sequence[str], *, refresh: bool = False) -> Mater """Classify every task without executing or committing anything. Not subject to the dirty-tree refusal: reading the state of a project - before deciding what to commit is what this is for. The walk is - topological because a task can only be classified after everything - upstream of it, and an upstream already classified as would-run is - passed down as ``None`` — check mode cannot know whether a rebuild - comes out byte-identical, and assuming it will would under-report. + before deciding what to commit is what this is for. Args: root: The project root. targets: What to classify; empty means everything. - refresh: Whether an output that is merely behind would be remade. + refresh: Whether a result that is merely behind would be remade. Returns: - ``planned`` naming each output that would run and why, ``behind`` + ``planned`` naming each result that would run and why, ``behind`` those made under an earlier environment, and ``current`` the rest. Raises: - ProjectError: If the spec, the universes or the lock cannot be - read, or a target matches nothing. + ProjectError: If the analysis or the lock cannot be read, or a + target matches nothing. """ report = MaterializeReport() - for key, verdict, _, _ in _classified(root, targets, report, refresh=refresh): - name = _name(key) + graph, environment, _ = _graph(root, targets, report) + for found in _classify(root, graph, environment, refresh=refresh): + verdict = found.decided.verdict if verdict.calls_for_a_remake(refresh=refresh): - report.planned[name] = verdict.why + report.planned[found.task.key] = verdict.why + elif found.upstream: + # Nothing here can know whether a rebuild comes out + # byte-identical, and assuming it will would under-report. + report.planned[found.task.key] = ( + f"`{found.upstream}`, which it reads, would be remade" + ) elif verdict.status == "behind": - report.behind[name] = verdict.why + report.behind[found.task.key] = verdict.why else: - report.current.append(name) + report.current.append(found.task.key) return report -def _classified( - root: Path, targets: Sequence[str], report: MaterializeReport, *, refresh: bool -) -> list[tuple[Key, assets.Verdict, assets.Manifest | None, dataset.LastWrite | None]]: - """Classify every task in topological order, reading nothing but disk. - - The walk both read-only modes share, so there is one answer to "what - is this output" and not one per verb. Topological because a task can - only be classified after everything upstream of it, and an upstream - already decided to run is passed down as ``None`` — nothing here can - know whether a rebuild comes out byte-identical, and assuming it will - would under-report. +@dataclass(frozen=True) +class _Classified: + """One task, and what the project's state says of its result.""" + + task: Task + decided: worker.Decided + #: The record at the result's first place, where it has one of its own. + record: Record | None + #: The commit that last wrote the result's files, if not its own run. + foreign: dataset.LastWrite | None + #: A task upstream of this one that may run, or empty. Only once it + #: has landed can anything say whether this result still stands. + upstream: Key + #: ASTRA finds no valid record, so no upstream outcome can spare it. + missing: bool + + def may_run(self, *, refresh: bool) -> bool: + """Whether a run has to hand this task to a worker.""" + return bool(self.upstream) or self.decided.verdict.calls_for_a_remake(refresh=refresh) + + +def _classify( + root: Path, graph: Graph, environment: str, *, refresh: bool +) -> list[_Classified]: + """Classify every task of *graph*, upstream first, reading nothing but + the project. + + The walk every mode shares, so there is one answer to "what is this + result" and not one per verb: ASTRA's checker is asked once, and each + task's findings go through the one classification rule, with the + history only the driver can read. Args: root: The project root. - targets: What to classify; empty means everything. - report: Collects the lock scan's warnings, and the note about - inputs this clone has not fetched. - refresh: Whether behind outputs count as running, which is what - decides whether their dependents see the sentinel. + graph: The tasks to classify, each after the tasks it reads. + environment: The IRI of the project's environment. + refresh: Whether behind results count as running, which is what + decides whether their dependents may run too. Returns: - One ``(key, verdict, manifest, foreign write)`` per task, upstream - first. + One entry per task, in the graph's order. """ - graph, env_version, _ = _graph(root, targets, report) - unfetched: set[str] = set() - classified = _classify_graph( - root, graph, env_version, refresh=refresh, - versions=assets.Versions(), unfetched=unfetched, - ) - if unfetched: - report.warnings.append( - "reported as out of date because their content is not in this " - f"clone, not because they changed: {', '.join(sorted(unfetched))}. " - "`lc materialize` fetches declared inputs before executing " - "recipes. Compute is required for outputs whose inputs cannot yet be checked." - ) - return classified - - -def _classify_graph( - root: Path, graph: Graph, env_version: str, *, refresh: bool, - versions: assets.Versions, unfetched: set[str], -) -> list[tuple[Key, assets.Verdict, assets.Manifest | None, dataset.LastWrite | None]]: - """Predict which outputs may run, using the same walk for checks and admission. - - Dependents of an output that may run are conservatively included: only the - worker can know whether rebuilding that input actually changed its bytes. - """ - would_run: set[Key] = set() + checked = assets.check(root) + may_run: set[Key] = set() classified = [] - for key in graph.order(): - task = graph.tasks[key] - manifest = assets.read(task.manifest_path) + for key, task in graph.tasks.items(): + found = checked[task.places[0].record] + record = found[0].record # History is git's to answer, so it enters the one classification - # rule as a value — computed here, where git lives, exactly as - # the worker's driver computes it for a run. - foreign = None if manifest is None else _foreign_write(root, task) - verdict = assets.classify( - definition_version=task.definition_version, - env_version=env_version, - manifest=manifest, - inputs=_predicted(root, task, would_run, versions, unfetched), - foreign=foreign, + # rule as a value — computed here, where git lives. + foreign = None if record is None else _foreign_write(root, task) + entry = _Classified( + task, + worker.decide(found, environment, foreign), + record, + foreign, + upstream=next((dep for dep in task.depends_on if dep in may_run), ""), + missing=any(status.record_status == "missing" for status in found), ) - if verdict.calls_for_a_remake(refresh=refresh): - would_run.add(key) - classified.append((key, verdict, manifest, foreign)) - + if entry.may_run(refresh=refresh): + may_run.add(key) + classified.append(entry) return classified -def _predicted( - root: Path, - task: Task, - would_run: set[Key], - versions: assets.Versions, - unfetched: set[str], -) -> dict[str, str | None]: - """Each input's version as check mode can know it. - - Args: - root: The project root, for naming a path in the report. - task: The output being classified. - would_run: Task keys already decided to be remade. - versions: The run's content-hash memo. - unfetched: Collects declared inputs whose content is not local, so - the report can say why they read as out of date. - - Returns: - Each input's version as check mode can know it: ``None`` for - anything that will be rebuilt, is absent, or cannot be read. - """ - predicted: dict[str, str | None] = {} - for name, path in task.inputs.items(): - upstream = task.produced_by.get(name) - if upstream is not None: - # The upstream's *recorded* digest, for the same reason a worker - # returns it rather than rehashing: on a clone that has fetched - # no annex content the files are dangling symlinks, and hashing - # them would report a different output and cascade a rebuild - # over a project that is perfectly up to date. - manifest = ( - None if upstream in would_run else assets.read(assets.manifest_path(path)) - ) - predicted[name] = manifest.data_version if manifest else None - elif not path.exists(): - predicted[name] = None - else: - try: - predicted[name] = versions.of(path) - except assets.ContentNotFetchedError: - # Not "it changed" — "I cannot tell". Conservative, and - # said out loud, because reporting a rebuild for a clone - # that simply has not fetched its inputs is misleading on - # its own. - unfetched.add(plan.declared_path(root, path)) - predicted[name] = None - except OSError: - # Unreadable for any other reason — a broken symlink inside - # the directory, a permission wall. Same answer as an input - # that is not there at all: it will be remade, and the - # recipe is where that failure belongs, with a real error. - predicted[name] = None - return predicted - - # ============================================================================= # Status — what the project holds, and where each of it came from # ============================================================================= @@ -285,7 +230,7 @@ def _predicted( class OutputStatus: """One output: what it is now, and the commit it came from.""" - #: ``universe/output_id``. + #: The result's task, as lc names it (``fit#baseline``). output: str status: assets.Status #: Why, for ``stale`` and ``behind``. Empty for ``current``. @@ -297,10 +242,10 @@ class OutputStatus: git_sha: str #: Its content identity, or empty if it was never materialized. data_version: str - #: Empty when the commit that last touched the output's directory is - #: its own run record. Otherwise that foreign commit's sha — the - #: agent-forged-file fact, which also makes the output read `stale`: - #: the manifest no longer describes the bytes, and a skip would + #: Empty when the commit that last touched the result's files is its + #: own run. Otherwise that foreign commit's sha — the + #: agent-forged-file fact, which also makes the result read `stale`: + #: the record no longer describes the bytes, and a skip would #: return the recorded digest forever. The sha rather than prose, #: so a machine consumer gets what `why`'s sentence cannot carry. #: History-based on purpose: it costs no hashing and answers on a @@ -320,7 +265,7 @@ def as_dict(self) -> dict[str, Any]: @dataclass class StatusReport: - """Every output the spec declares, in dependency order.""" + """Every result the analysis plans, in dependency order.""" outputs: list[OutputStatus] = field(default_factory=list) #: The lock scan's prose, and inputs this clone has not fetched. @@ -367,9 +312,9 @@ def as_dict(self) -> dict[str, Any]: def status(root: Path) -> StatusReport: - """Report what state every declared output is in. + """Report what state every planned result is in. - Reads manifests and hashes declared inputs; runs nothing, commits + Reads records and hashes declared inputs; runs nothing, commits nothing, and does not care whether the tree is clean. Classified with ``refresh=False``, because this says what the project *is* rather than what some run would do to it. @@ -378,11 +323,10 @@ def status(root: Path) -> StatusReport: root: The project root. Returns: - One record per declared output, upstream first. + One record per planned result, upstream first. Raises: - ProjectError: If the spec, the universes or the lock cannot be - read. + ProjectError: If the analysis or the lock cannot be read. """ report = MaterializeReport() result = StatusReport() @@ -392,17 +336,18 @@ def status(root: Path) -> StatusReport: result.image = {"tag": tag, "state": state, "archive": archive} result.sandbox = _sandbox_line(result.mode) stamps = [] - for key, verdict, manifest, foreign in _classified(root, [], report, refresh=False): - if manifest and manifest.finished_at: - stamps.append(manifest.finished_at) + graph, environment, _ = _graph(root, [], report) + for found in _classify(root, graph, environment, refresh=False): + if found.record and found.record.finished: + stamps.append(found.record.finished) result.outputs.append( OutputStatus( - output=_name(key), - status=verdict.status, - why=verdict.why, - git_sha=manifest.git_sha if manifest else "", - data_version=manifest.data_version if manifest else "", - foreign_write=foreign.sha if foreign else "", + output=found.task.key, + status=found.decided.verdict.status, + why=found.decided.verdict.why, + git_sha=found.record.commit if found.record else "", + data_version=found.record.content_hash if found.record else "", + foreign_write=found.foreign.sha if found.foreign else "", ) ) result.crate = _crate_line(root, max(stamps, default="")) @@ -414,16 +359,16 @@ def _crate_line(root: Path, newest: str) -> str: """One line placing the publication view, from repository facts alone. Lag is read off the document itself, not history: the render pins - ``datePublished`` to the newest manifest ``finished_at``, so a date - that no longer matches the manifests — in either direction, a rerun + ``datePublished`` to the newest record's ``finished``, so a date + that no longer matches the records — in either direction, a rerun adds an output the view predates and a dropped output regresses the newest — means the view no longer describes the outputs. That is the line's whole claim, and it is worded to it: a crate-affecting edit - that moves no manifest (the lock's bytes, the spec) is invisible + that moves no record (the lock's bytes, the spec) is invisible here, and fine — the next materialize converges those anyway, where a rerun's lag has no other surface. No rocrate import (the crate is the one materialize-only dependency on status's path) and no git: - the manifests were already read by the walk. + the records were already read by the walk. """ spdx = project.license_of(root) path = root / project.CRATE_FILENAME @@ -446,29 +391,32 @@ def _crate_line(root: Path, newest: str) -> str: def _foreign_write(root: Path, task: Task) -> dataset.LastWrite | None: - """Find the commit that last touched *task*'s manifest, unless it is - the output's own run record — then ``None``, the clean answer. The - fact only: the verdict's prose is `classify`'s, like every other - why. - - Both paths: a forged *result* is the case the check exists for, and a - doctored *manifest* is the other half — while the manifest is also - what carries the history across a re-declared format, where the - payload's own path is new and would read as clean. + """Find the commit that last touched *task*'s files, unless it is the + result's own run — then ``None``, the clean answer. The fact only: + the verdict's prose is `classify`'s, like every other why. + + Both files at every place: a forged *result* is the case the check + exists for, and a doctored *record* is the other half — while the + record is also what carries the history across a re-declared format, + where the payload's own path is new and would read as clean. """ - write = dataset.last_writer(root, task.output_path, task.manifest_path) + write = dataset.last_writer(root, *_files(task)) if not write or write.subject == datalad_run_subject(task.key): return None return write +def _files(task: Task) -> list[str]: + """The bytes and the record at each of *task*'s places.""" + return [path for place in task.places for path in (place.bytes, place.record)] + + def _sandbox_line(mode: str) -> str: """One line naming the enforcement a run on this host would get. The prose restates each backend's constant attestation, because `Backend.attest` needs a built policy and a status header must not - build one. Keep it in step with the `attest` implementations — the - manifests, which record the real thing, are always authoritative. + build one. Keep it in step with the `attest` implementations. """ if mode == "containerized": if runtime := container.runtime_hint(): @@ -522,7 +470,7 @@ def materialize( # before the image: a refusal here must not cost a minutes-long # build, and must not leave an archive commit behind a run that # "failed" on a typo in the spec. - graph, env_version, full = _graph(root, targets, report) + graph, environment, full = _graph(root, targets, report) dsid = dataset.dataset_id(root) if not graph.tasks: # No tasks is not no project: a spec whose outputs were all dropped @@ -530,14 +478,10 @@ def materialize( # maintainer. Nothing is submitted, so no allocation is needed. _converge_crate(root, report, full, dsid) return report - versions = assets.Versions() - classified = _classify_graph( - root, graph, env_version, refresh=refresh, versions=versions, unfetched=set(), - ) + classified = _classify(root, graph, environment, refresh=refresh) with cluster_for_run(cluster_id) as scheduler: requirements = scheduler.validate( - graph.tasks[key] for key, verdict, _, _ in classified - if verdict.calls_for_a_remake(refresh=refresh) + found.task for found in classified if found.may_run(refresh=refresh) ) _fetch_inputs(root, graph, report) # Materialize is one of the two verbs allowed to build the image (the @@ -552,54 +496,43 @@ def materialize( report.warnings.extend(f"uv: {w}" for w in container.converge(runtime)) # The run's driver-resolved facts, each read once: HEAD because the # driver commits as outputs land and a per-task read would stamp - # later manifests with a commit this run created; the uv probe - # because attestation is a fact about the run (and empty is an - # answer, not a failure); one content-hash memo because a declared - # input shared by several outputs is the same bytes every time. - for path in { - path - for task in graph.tasks.values() - for name, path in task.inputs.items() - if name not in task.produced_by - }: - try: - versions.of(path) - except Exception: - # Keep unreadable-input failures inside the tasks that need them. - pass - context = worker.RunContext( - env_version=env_version, - head=dataset.head(root), - versions=versions, - runtime=runtime, - uv_version=project.uv_version(root), - ) + # later records with a commit this run created. + commit, _ = dataset.head(root) + context = worker.RunContext(environment=environment, commit=commit, runtime=runtime) pending: dict[Key, Any] = {} handles = [] - # Current outputs are values, not Dask tasks: they need no allocation + # Results left alone are values, not Dask tasks: they need no allocation # resources. Futures for everything that may run retain dependency order. - for key, verdict, manifest, foreign in classified: - task = graph.tasks[key] - if not verdict.calls_for_a_remake(refresh=refresh): - assert manifest is not None and verdict.status != "stale" + for found in classified: + task, decided = found.task, found.decided + if not found.may_run(refresh=refresh): result = worker.TaskResult( - key, verdict.status, data_version=manifest.data_version, reason=verdict.why, + task.key, + decided.verdict.status, # type: ignore[arg-type] # stale may run + decided.content_hash, + decided.size, + reason=decided.verdict.why, ) - pending[key] = result + pending[task.key] = result _consume(root, task, result, dsid, runtime, report) continue - pending[key] = scheduler.submit( + pending[task.key] = scheduler.submit( worker.materialize, root, task, context, refresh, - foreign, + # Downstream of a remake, what the driver found may not + # survive it, so the task asks again once its upstreams have + # landed — unless it has no record, which nothing upstream + # can spare. + None if found.upstream and not found.missing else decided, + found.foreign, *[pending[dep] for dep in task.depends_on], - key=_name(key), - resources=requirements[key], + key=task.key, + resources=requirements[task.key], ) - handles.append(pending[key]) + handles.append(pending[task.key]) from lightcone.engine.compute import UNSTOPPED # An unreported task can still have a running subprocess. Leave its @@ -614,15 +547,15 @@ def materialize( raise raise ProjectError(f"{exc}. {UNSTOPPED}") from exc # The tree was clean at the start-of-run refusal and save/restore - # keeps `results/` clean, so anything dirty *now* was edited while - # the graph ran — and every manifest records the starting commit, - # which no longer describes that code. A warning, never a manifest + # keeps results clean, so anything dirty *now* was edited while + # the graph ran — and every record names the starting commit, + # which no longer describes that code. A warning, never a record # field: the driver does not rewrite files the worker owns. if edited := dataset.status(root): names = ", ".join(sorted(path for _, path in edited)) report.warnings.append( - f"edited while the run was in flight: {names} — the manifests " - "record the starting commit, which no longer describes this code" + f"edited while the run was in flight: {names} — the records " + "name the starting commit, which no longer describes this code" ) _converge_crate(root, report, full, dsid) return report @@ -637,7 +570,7 @@ def _consume( report: MaterializeReport, ) -> None: """Record one finished task, and commit or undo what it left on disk.""" - name = _name(task.key) + name = task.key if result.notes: # Named on a line of their own rather than prefixed onto each: a # prefix would land in the middle of a multi-line remedy and make @@ -645,7 +578,7 @@ def _consume( report.notes.extend([f"{name}:", *result.notes]) if result.status == "ok": - dataset.save(root, _owned(root, task), run_record(root, task, dsid, runtime)) + dataset.save(root, _owned(task), run_record(root, task, dsid, runtime)) report.made.append(name) return @@ -657,7 +590,7 @@ def _consume( report.behind[name] = result.reason return - dataset.restore(root, _owned(root, task)) + dataset.restore(root, _owned(task)) getattr(report, result.status).append(name) report.warnings.append(f"{name}: {result.reason}") @@ -716,7 +649,7 @@ def validate(self, tasks: Iterable[Task]) -> dict[Key, dict[str, float]]: try: requests[task.key] = TaskResources.parse(task.resources).requirements(capacities) except ProjectError as exc: - raise ProjectError(f"{_name(task.key)}: {exc}") from exc + raise ProjectError(f"{task.key}: {exc}") from exc return requests def submit(self, fn: Any, *args: Any, key: str, resources: dict[str, float]) -> Any: @@ -736,7 +669,7 @@ def completed(self, handles: list[Any]) -> Iterator[worker.TaskResult]: for _, result in as_completed( # type: ignore[no-untyped-call] handles, with_results=True, loop=self.client.loop ): - if not self.output.wait(_name(result.key)): + if not self.output.wait(result.key): result = replace( result, notes=(*result.notes, @@ -775,12 +708,9 @@ def _fetch_inputs(root: Path, graph: Graph, report: MaterializeReport) -> None: lc fetches rather than telling anyone to — the storage invariant — and only here: ``--check`` and ``status`` are read-only verbs that - must not start network transfers, so there an unfetched input stays a - reported fact. In-tree inputs only, because an absolute input outside - the repository has no annex to fetch it from (the recorded weaker - promise). Unconditional rather than probed: ``git annex get`` on - content already present is a fast no-op, and one batch invocation - beats a detection walk. + must not start network transfers. Unconditional rather than probed: + ``git annex get`` on content already present is a fast no-op, and one + batch invocation beats a detection walk. A failed fetch is a *warning*, never a refusal: independent tasks still run, and the task whose input is genuinely unreachable reports @@ -789,10 +719,10 @@ def _fetch_inputs(root: Path, graph: Graph, report: MaterializeReport) -> None: """ declared = sorted( { - plan.declared_path(root, path) + path for task in graph.tasks.values() - for name, path in task.inputs.items() - if name not in task.produced_by and root in path.parents + for entry, path in task.inputs.items() + if entry not in task.produced_by } ) if not declared: @@ -854,22 +784,22 @@ def _converge_crate(root: Path, report: MaterializeReport, full: Graph, dsid: st ) return # A set difference rather than a walk driven by the spec: what is on - # disk is exactly what the *previous* spec declared, so the edit that - # orphans a manifest is the same edit that drops it from the graph. + # disk is exactly what the *previous* spec planned, so the edit that + # orphans a record is the same edit that drops it from the graph. # Enumerated from git, which answers for the tree as committed. - expected = {task.manifest_path for task in full.tasks.values()} - for manifest in sorted(dataset.tracked_manifests(root)): - if manifest not in expected and assets.read(manifest) is not None: - report.warnings.append( - f"{plan.declared_path(root, manifest)} is a manifest the spec no " - "longer declares, so its output is not in the publication view" - ) + expected = {place.record for task in full.tasks.values() for place in task.places} + for record in sorted(set(dataset.tracked_records(root)) - expected): + report.warnings.append( + f"{record} is the record of a result the analysis no longer plans, " + "so that result is not in the publication view" + ) try: document = crate.render( root, full, license=spdx, dsid=dsid, + remote=dataset.head(root)[1], writer=functools.partial(dataset.last_writer, root), keys=dataset.annex_keys(root), ) @@ -929,15 +859,12 @@ def run_record(root: Path, task: Task, dsid: str, runtime: container.Runtime) -> # form of the requirement contains spaces. "cmd": ( f"uv run --no-project --with '{_engine_requirement()}' -- " - f"python -m lightcone.engine.worker {task.universe_id}/{task.output_id}" + f"python -m lightcone.engine.worker {shlex.quote(task.key)}" ), "dsid": dsid, "exit": 0, - "inputs": sorted(plan.declared_path(root, path) for path in task.inputs.values()), - "outputs": [ - plan.declared_path(root, task.output_path), - plan.declared_path(root, task.manifest_path), - ], + "inputs": sorted(set(task.inputs.values())), + "outputs": _files(task), "pwd": ".", } if runtime.mode == "containerized": @@ -957,11 +884,11 @@ def datalad_run_subject(key: Key) -> str: ours to adjust. One function, shared with the foreign-write check: an output is - cleanly written iff the commit that last touched its directory + cleanly written iff the commit that last touched its files carries exactly this subject, so the composer and the comparator must not be two strings that can drift apart. """ - return f"[DATALAD RUNCMD] {key[1]} [{key[0]}]" + return f"[DATALAD RUNCMD] {key}" def _engine_requirement() -> str: @@ -1045,51 +972,28 @@ def _graph( + "; ".join(scan.machine_config) ) - env_version = identity.env_version(root) + environment = identity.environment(root) full = plan.build(root) + report.warnings.extend(full.notes) graph = full.closure(full.resolve(list(targets))) if targets else full - - # A declared input outside the project is hashed into the manifest like - # any other, so a change to it still cascades — but it is not in the - # repository, so the commit that records the output cannot bring it - # back. That is a weaker promise than the rest of the layer makes, and - # the only honest thing to do about it is say so. - outside = { - plan.declared_path(root, path) - for task in graph.tasks.values() - for name, path in task.inputs.items() - if name not in task.produced_by and root not in path.parents - } - if outside: - report.warnings.append( - "declared inputs outside the project are recorded by content but " - "not stored in it, so a commit cannot restore them: " - + ", ".join(sorted(outside)) - ) - return graph, env_version, full + return graph, environment, full -def _owned(root: Path, task: Task) -> list[str]: - """Every path one output owns, as git pathspecs. +def _owned(task: Task) -> list[str]: + """Every path one result owns, as git pathspecs. - Globs rather than the two literal paths, and each earns its magic. The - payload's covers whatever format the *previous* run wrote, so a - re-declared serialization stages the old file's deletion instead of - leaving it behind untracked. The manifest's trailing ``*`` covers the - ``.tmp`` a failed write leaves, which nothing else would sweep. + A glob for the payload rather than its literal path: it covers + whatever format the *previous* run wrote, so a re-declared + serialization stages the old file's deletion instead of leaving it + behind untracked. Never the parent directory: siblings share it, and under Dask they are in flight — ``git clean`` on it would delete a running task's bytes. """ - parent = plan.declared_path(root, task.output_path.parent) - return [ - f":(glob){parent}/{task.output_id}.*", - f":(glob){parent}/.{task.output_id}{assets.MANIFEST_SUFFIX}*", - ] - - -def _name(key: Key) -> str: - return f"{key[0]}/{key[1]}" + owned = [] + for place in task.places: + owned += [f":(glob){posixpath.dirname(place.bytes)}/{place.name}.*", place.record] + return owned def _dirty(root: Path, changes: Sequence[tuple[str, str]]) -> str: @@ -1097,11 +1001,12 @@ def _dirty(root: Path, changes: Sequence[tuple[str, str]]) -> str: Two path classes, because they call for opposite actions: work the researcher owns has to be committed, and anything under ``results/`` - is lc's to write, so a change there is wreckage to discard rather than - a contribution to keep. + or ``.astra/`` is lc's to write, so a change there is wreckage to + discard rather than a contribution to keep. """ - theirs = [c for c in changes if not c[1].startswith("results/")] - ours = [c for c in changes if c[1].startswith("results/")] + written = ("results/", ".astra/") + theirs = [c for c in changes if not c[1].startswith(written)] + ours = [c for c in changes if c[1].startswith(written)] lines = [ f"uncommitted changes in {root} — every materialization is committed " @@ -1115,12 +1020,13 @@ def _dirty(root: Path, changes: Sequence[tuple[str, str]]) -> str: *(f" {code.strip() or '??'} {path}" for code, path in theirs), ] if ours: + where = " ".join(sorted({path.partition("/")[0] + "/" for _, path in ours})) lines += [ "", " if a cluster run was interrupted, stop its allocation first:", " lc compute down ", - " confirm its recipes have stopped, then discard these (lc writes results/):", - " git restore --staged --worktree results/ && git clean -fd results/", + " confirm its recipes have stopped, then discard these (lc writes them):", + f" git restore --staged --worktree {where} && git clean -fd {where}", *(f" {code.strip() or '??'} {path}" for code, path in ours), ] return "\n".join(lines) diff --git a/src/lightcone/engine/plan.py b/src/lightcone/engine/plan.py index 513f410c..8b34b055 100644 --- a/src/lightcone/engine/plan.py +++ b/src/lightcone/engine/plan.py @@ -1,70 +1,70 @@ """The spec, read as a graph of tasks. -``astra.yaml`` × ``universes/*.yaml`` gives one task per -``(universe, output)`` pair that has a recipe. A task carries everything -executing it needs and nothing about *how* it will be executed: the -rendered command, where its bytes go, what it reads, which decisions it -was made under, and its ``definition_version``. - -What the spec *means* is ASTRA's to say. ``astra.resolve`` settles each -universe's decisions, resolves every output's inputs to what supplies -them, drops the outputs whose ``when:`` does not hold, and renders the -recipe grammar — so scoping, ``from:`` references and sub-analysis -nesting are read here rather than re-derived. - -Where an output lands is composed, never chosen: an output is the single -file ``results//.``, and the format comes from the -spec — so the whole contents of ``results/`` are a pure function of it. - -Nothing here schedules anything. Ordering is Dask's job at execution time -and a topological walk's job in ``--check``; this module only says which -task depends on which. +ASTRA plans the results of an analysis — one per output built once, one +per universe for an output built per universe — and says of each where +its bytes and its record go, what it reads, the options its decisions +select and the command that makes it. A task is one planned result with +everything executing it needs and nothing about *how* it will be +executed. + +What the spec *means* is ASTRA's to say: ``astra.api.plan`` validates the +stack, settles every universe's selection, follows declared analyses and +fills the recipe grammar, so none of it is re-derived here. This module +only carries each result across to the workers as plain data, and +refuses what this engine cannot execute. + +Nothing here schedules anything. The plan lists each result after every +result it reads, and that order is kept; at execution time ordering is +Dask's job. """ from __future__ import annotations -from dataclasses import dataclass, field -from graphlib import CycleError, TopologicalSorter +import posixpath +from collections.abc import Iterator +from contextlib import contextmanager +from dataclasses import asdict, dataclass, field from pathlib import Path -from typing import Any +from typing import TYPE_CHECKING, Any -from lightcone.engine import assets, identity from lightcone.engine.project import SPEC_FILENAME, ProjectError -#: A task's identity within a run: which universe, which output. -Key = tuple[str, str] +if TYPE_CHECKING: + from astra.diagnostics import Diagnostic + from astra.plan import Result + from astra.universes import Place + +#: A task's identity within a run: its result's address, after the folder +#: of the analysis the result belongs to when that is not the project's own +#: (``fit#baseline``, ``stage/fit#v1``). +Key = str @dataclass(frozen=True) class Task: - """One output, in one universe: everything needed to make it.""" - - universe_id: str - output_id: str - #: The single file this output is. - output_path: Path - #: The recipe with its placeholders substituted — a shell command. - recipe: str - #: Declared input name → the path it resolves to. Upstream outputs are - #: their files; everything else is whatever ``source`` named. - inputs: dict[str, Path] - #: The subset of ``inputs`` another task produces, and which one. + """One planned result: everything needed to make it.""" + + key: Key + #: Where the result's record and bytes go, from the project root, one + #: per name ASTRA gives it. The recipe writes the first; the others + #: receive copies. + places: tuple[Place, ...] + #: The recipe with its placeholders filled — a shell command, its paths + #: relative to ``workdir``. + command: str + #: The folder the recipe runs in, from the project root: its output's + #: analysis's. + workdir: str + #: Each entry of the output's ``inputs``, as written and in order → the + #: file it reads, from the project root. + inputs: dict[str, str] + #: The entries of ``inputs`` another task produces, and which one. produced_by: dict[str, Key] + #: Each entry of the output's ``decisions`` → the option selected. decisions: dict[str, str] - definition_version: str #: ASTRA's declaration; executor support is checked only when executing. resources: dict[str, Any] = field(default_factory=dict) - @property - def manifest_path(self) -> Path: - """This output's manifest sidecar.""" - return assets.manifest_path(self.output_path) - - @property - def key(self) -> Key: - """This task's identity within a run.""" - return (self.universe_id, self.output_id) - @property def depends_on(self) -> tuple[Key, ...]: """The tasks that must be made first, deduplicated, in order.""" @@ -73,28 +73,12 @@ def depends_on(self) -> tuple[Key, ...]: @dataclass(frozen=True) class Graph: - """Every task a run could make, and how they relate.""" + """Every task a run could make, each after the tasks it reads.""" tasks: dict[Key, Task] - - def order(self) -> list[Key]: - """Return the tasks in dependency order. - - Only ``--check`` needs this, to classify a task after everything - upstream of it. Execution never calls it: Dask derives the same - order from the futures it is handed. - - Returns: - Every task key, dependencies first. - - Raises: - ProjectError: If the outputs depend on each other in a cycle. - """ - sorter = TopologicalSorter({k: set(t.depends_on) for k, t in self.tasks.items()}) - try: - return list(sorter.static_order()) - except CycleError as e: - raise ProjectError(f"the outputs depend on each other in a cycle: {e.args[1]}") from e + #: What ASTRA notes about the plan that a run would silently live with: + #: an output nothing plans, a command reaching outside its analysis. + notes: tuple[str, ...] = () def closure(self, targets: list[Key]) -> Graph: """Narrow the graph to *targets* and what they depend on. @@ -106,7 +90,8 @@ def closure(self, targets: list[Key]) -> Graph: targets: The task keys asked for. Returns: - A graph holding *targets* plus their transitive dependencies. + A graph holding *targets* plus their transitive dependencies, + in the plan's order. """ wanted: set[Key] = set() pending = list(targets) @@ -122,8 +107,9 @@ def resolve(self, targets: list[str]) -> list[Key]: """Turn what a user typed into task keys. Args: - targets: Each an output id — matching every universe that has - it — or ``/`` for exactly one. + targets: Each a task as lc names it (``fit#baseline``), or a + result's name alone (``fit``) — matching it in every + universe that builds it. Returns: The matching task keys, in the order given. @@ -134,12 +120,13 @@ def resolve(self, targets: list[str]) -> list[Key]: """ keys: list[Key] = [] for target in targets: - universe, _, output = target.rpartition("/") matched = [ - key for key in self.tasks if key[1] == output and universe in ("", key[0]) + key + for key, task in self.tasks.items() + if target == key or any(target == place.name for place in task.places) ] if not matched: - known = ", ".join(sorted(f"{u}/{o}" for u, o in self.tasks)) or "none" + known = ", ".join(self.tasks) or "none" raise ProjectError(f"no output matches `{target}`. Available: {known}") keys.extend(matched) return keys @@ -151,204 +138,155 @@ def resolve(self, targets: list[str]) -> list[Key]: def build(root: Path) -> Graph: - """Read a project's spec and universes into a graph of tasks. + """Read a project's analysis into a graph of tasks. Args: root: The project root. Returns: - One task per ``(universe, output)`` pair that has a recipe and is - active in that universe. + One task per planned result this engine builds, each after the + tasks it reads. Raises: - ProjectError: If the spec is missing, declares no universe, gives - two universes the same id, or names an input nothing provides. + ProjectError: If there is no analysis, ASTRA finds it invalid, or + it plans a result this engine cannot build. """ - from astra.helpers import load_yaml, resolve_analysis_tree + from astra import api + from astra.plan import render_command + from astra.universes import Place - spec_path = root / SPEC_FILENAME - if not spec_path.is_file(): - raise ProjectError( - f"{root}: no {SPEC_FILENAME} — there is no analysis to materialize." - ) - universes = sorted((root / "universes").glob("*.yaml")) - if not universes: - raise ProjectError( - f"{root}/universes/ declares no universe — a run needs at least one " - "set of decisions to make outputs under." - ) - _validate(spec_path, universes) - spec = dict(resolve_analysis_tree(load_yaml(spec_path), root)) + with asking_astra(root): + planned = api.plan(root) + # ASTRA measures every path from the top of the repository; everything + # lc does is measured from the project root, which may be below it. + above = posixpath.dirname(planned.root) + + def key_of(result: Result) -> Key: + return posixpath.join(relative(result.owner.folder, above), result.places[0].address) + keys = {(result.output, result.universe): key_of(result) for result in planned.results} tasks: dict[Key, Task] = {} - declared_in: dict[str, Path] = {} - for path in universes: - universe = load_yaml(path) - universe_id = str(universe.get("id") or path.stem) - # A universe id names a directory under results/, so two files - # claiming one would write to the same place — and since the second - # simply replaces the first here, the outputs of one of them would - # go missing with nothing said. - if (first := declared_in.get(universe_id)) is not None: + for result in planned.results: + if result.read_only: + continue + key = keys[result.output, result.universe] + from_commit = [read.entry for read in result.inputs if read.commit is not None] + if result.run_at is not None or from_commit: + what = f"reads `{from_commit[0]}` from" if from_commit else "belongs to" raise ProjectError( - f"{first.name} and {path.name} both declare the universe " - f"`{universe_id}`, so both would materialize into " - f"results/{universe_id}/. Give each universe its own id." + f"`{key}` {what} an analysis declared with a `commit`, which lc does not " + "build from. Declare that analysis by `path` alone to build it from the " + "working tree." ) - declared_in[universe_id] = path - for task in _tasks(root, universe_id, spec, universe): - tasks[task.key] = task - - return Graph(tasks=tasks) - - -def declared_path(root: Path, path: Path) -> str: - """Name *path* the way the analysis declared it. + owner = relative(result.owner.folder, above) + if any((root / parent / ".git").exists() for parent in _folders(owner)): + raise ProjectError( + f"`{key}` belongs to the analysis in {owner}, which is a " + "repository of its own — lc commits results in the project's repository " + "only. Run lc from that analysis instead." + ) + tasks[key] = Task( + key=key, + places=tuple( + Place( + place.name, + place.address, + relative(place.record, above), + relative(place.bytes, above), + ) + for place in result.places + ), + command=render_command(result, root / relative(result.workdir, above)), + workdir=relative(result.workdir, above), + inputs={read.entry: relative(read.path, above) for read in result.inputs}, + produced_by={ + read.entry: keys[read.result.output, read.result.universe] + for read in result.inputs + if read.result is not None + }, + decisions={k: v for k, v in result.decisions.items() if v is not None}, + resources=( + {k: v for k, v in asdict(result.resources).items() if v is not None} + if result.resources + else {} + ), + ) + notes = tuple( + note.message + for note in planned.notes + if note.built and note.kind in ("unplanned", "outside_folder") + ) + return Graph(tasks=tasks, notes=notes) - Project-relative inside the tree, absolute outside it — the two forms a - recipe, a manifest and a ``[DATALAD RUNCMD]`` record all want, and the - reason a declared input with an absolute ``source:`` has somewhere to - be written down rather than being a crash. - Deliberately **not** resolved. Every declared input under ``data/`` is - an annex symlink, so resolving would name - ``.git/annex/objects/SHA256E-…`` — the storage rather than the input, - and a path no one can fetch. +def relative(path: str, above: str) -> str: + """Measure a path ASTRA gives from the project root. Args: - root: The project root. - path: What to name. + path: A path from the top of the repository; empty for the top. + above: The project root's own path from that top; empty when the + project is the repository. Returns: - A POSIX path, relative to *root* where it is under it. + The same file's path from the project root; empty for the root. """ - try: - return path.relative_to(root).as_posix() - except ValueError: - return path.as_posix() + found = posixpath.relpath(path or ".", above or ".") + return "" if found == "." else found -def _validate(spec_path: Path, universes: list[Path]) -> None: - """Refuse a spec ASTRA rejects, before anything is resolved. +def _folders(folder: str) -> Iterator[str]: + """*folder* and each folder above it, below the project root.""" + while folder and not folder.startswith(".."): + yield folder + folder = posixpath.dirname(folder) - Resolution answers what a valid spec *means*; it does not re-check - that it is one. So an invalid spec reaches it as a missing decision or - an unresolvable input — blaming the run for a fault in the file, and - at a point far from the line that caused it. Asking ASTRA first costs - one pass over a spec file and moves the error to where it can be - fixed. + +@contextmanager +def asking_astra(root: Path) -> Iterator[None]: + """Turn what ASTRA raises about a project into a ``ProjectError``. + + One place, because every question lc puts to ASTRA — the plan, the + check — can fail the same ways, and each must reach the user as the + same clean refusal rather than as another package's traceback. Args: - spec_path: The project's ``astra.yaml``. - universes: Every universe file that will be resolved against it. + root: The project root, for naming a missing analysis. Raises: - ProjectError: Listing every structural and semantic error found, - in ASTRA's own words. + ProjectError: If there is no analysis, it breaks one of ASTRA's + rules, or git or ASTRA's cache cannot be used. """ - from astra.validation import ( - validate_analysis_file, - validate_analysis_schema, - validate_universe_file, - ) - - problems = [ - *validate_analysis_schema(spec_path), - *(str(e) for e in validate_analysis_file(spec_path)), - ] - for path in universes: - problems += [f"{path.name}: {e}" for e in validate_universe_file(path, spec_path)] - if problems: - listed = "\n".join(f" {problem}" for problem in problems) - raise ProjectError( - f"{spec_path} does not validate, so there is nothing to materialize " - f"from it:\n{listed}" - ) + from astra import api + from astra.cache import CacheError + from astra.sources.git import GitError + if not (root / SPEC_FILENAME).is_file(): + raise ProjectError(f"{root}: no {SPEC_FILENAME} — there is no analysis to materialize.") + try: + yield + except api.InvalidAnalysisError as e: + raise ProjectError(invalid(root, e.report.diagnostics)) from e + except (api.NoAnalysisError, GitError, CacheError, OSError) as e: + raise ProjectError(str(e)) from e -def _tasks( - root: Path, - universe_id: str, - spec: dict[str, object], - universe: dict[str, object], -) -> list[Task]: - """Every task one universe contributes. - ``resolve_outputs`` has already dropped what this universe does not - produce, so the only filter left is whether an output carries a - command: a re-export names bytes another output makes, and making it - twice under two ids is not a thing to do. - """ - from astra.resolve import render_command, resolve_outputs - - resolved = resolve_outputs(spec, universe, root) - # The whole resolved output, not just its id: an upstream input has to - # resolve to the *file* another task writes, which needs that output's - # format. A set of ids cannot say it. - executable = {out.id: out for out in resolved if out.command} - - if absent := sorted(out.id for out in executable.values() if not out.format): - raise ProjectError( - f"{len(absent)} output(s) declare no `format:`: {', '.join(absent)}. " - "lc names each output's file from it, so there is nowhere to write them — " - "add the artifact's file extension, e.g. `format: png`." - ) +def invalid(root: Path, diagnostics: tuple[Diagnostic, ...]) -> str: + """Word the refusal of an analysis ASTRA rejects, in ASTRA's own words. - def file_of(out: object) -> Path: - return assets.output_path( - root, - universe_id, - str(out.id), # type: ignore[attr-defined] - str(out.format), # type: ignore[attr-defined] - ) + Args: + root: The project root. + diagnostics: What ASTRA reported; the errors are listed. - tasks = [] - for out in resolved: - if not out.command: - continue - output_path = file_of(out) - values: dict[str, str] = {} - paths: dict[str, Path] = {} - produced_by: dict[str, Key] = {} - for declared in out.inputs: - if declared.produced_by in executable: - produced_by[declared.id] = (universe_id, declared.produced_by) - paths[declared.id] = file_of(executable[declared.produced_by]) - elif declared.source: - # An absolute `source:` wins over the join — pathlib's own - # rule, and the one anyone writing one expects. - paths[declared.id] = root / declared.source - else: - raise ProjectError( - f"output `{out.id}` declares the input `{declared.id}`, but no " - "output produces it and no declared input gives it a source." - ) - values[declared.id] = declared_path(root, paths[declared.id]) - - try: - recipe = render_command( - out.command, - inputs=values, - decisions=out.decisions, - output=declared_path(root, output_path), - ) - except ValueError as e: - raise ProjectError(f"output `{out.id}`: {e}") from e - resources = dict((out.definition.get("recipe") or {}).get("resources") or {}) - - tasks.append( - Task( - universe_id=universe_id, - output_id=out.id, - output_path=output_path, - recipe=recipe, - inputs=paths, - produced_by=produced_by, - decisions=out.decisions, - definition_version=identity.definition_version( - recipe=recipe, decisions=out.decisions, fmt=str(out.format) - ), - resources=resources, - ) - ) - return tasks + Returns: + The message, one line per broken rule, each placed in its file. + """ + listed = "\n".join( + f" {d.location.file}:{d.location.line or 0}:{d.location.column or 0} " + f"{d.code} {d.message}" + for d in diagnostics + if d.level == "error" + ) + return ( + f"{root / SPEC_FILENAME} does not validate, so there is nothing to materialize " + f"from it:\n{listed}" + ) diff --git a/src/lightcone/engine/project.py b/src/lightcone/engine/project.py index ed56903c..752fa1eb 100644 --- a/src/lightcone/engine/project.py +++ b/src/lightcone/engine/project.py @@ -14,8 +14,6 @@ from pathlib import Path from typing import Literal -from astra.scaffold import create_boilerplate - from lightcone.engine import dataset, templates SPEC_FILENAME = "astra.yaml" @@ -204,17 +202,10 @@ def converge(directory: Path, *, write: bool = True) -> ConvergenceReport: if write: directory.mkdir(parents=True, exist_ok=True) - # `astra.scaffold` is astra's public scaffold API, the same one - # `astra init` delegates to: it writes the spec — an empty astra.yaml - # plus a universes/baseline.yaml selecting nothing, which converge as - # one item so the baseline never lands beside a user-authored spec — - # and nothing else. It is stdlib-only and imports in milliseconds, which is why it - # sits at module scope where the validation stack cannot. - c.item( - "astra.yaml", - (directory / SPEC_FILENAME).exists(), - lambda: create_boilerplate(directory), - ) + # The spec `astra init` writes, taken from astra rather than through + # its `init`: that one also makes the git repository, which here is + # the dataset item's to converge. + c.file("astra.yaml", directory / SPEC_FILENAME, _minimal_spec) _converge_uv_project(c, directory) c.file( ".gitignore", @@ -262,6 +253,12 @@ def converge(directory: Path, *, write: bool = True) -> ConvergenceReport: return c.report +def _minimal_spec() -> str: + from astra.api import MINIMAL + + return MINIMAL + + def require_uv() -> None: """Refuse early when uv is absent — it is the environment substrate. @@ -335,8 +332,8 @@ def sync(directory: Path) -> list[str]: Both entry points that execute recipes converge before they start, rather than checking and refusing: workers pass ``--no-sync``, so - nothing else would notice a lock edited without a sync, and a manifest - recording an environment the recipe did not run under is the identity + nothing else would notice a lock edited without a sync, and a record + naming an environment the recipe did not run under is the identity model saying something untrue. The containerized twin is ``container.sync``, which runs the same uv command inside the image — callers pick by :func:`mode`, because only they know whether an image @@ -395,7 +392,7 @@ def _converge_dataset(c: _Converger, directory: Path) -> None: itself, so an enclosing repository that already has one is adopted. Then the two files that make the storage policy: ``.gitattributes`` - routes results and inputs into the annex and keeps manifests in git, + routes results and inputs into the annex and keeps the rest in git, and ``.datalad/config`` carries a dataset id — the one thing a git + git-annex repository lacks to *be* a DataLad dataset, so a project is one from birth rather than by later adoption. The id is generated @@ -792,29 +789,6 @@ def uv_scrub_warning() -> str: return "" -def uv_version(directory: Path) -> str: - """Ask uv its version, for the manifest's attestation. - - Read once per run by whoever owns the run — the driver, or the rerun - entry point — and handed down, the HEAD discipline. Empty on any - failure: attestation must never fail a run. - - Args: - directory: Where to run the probe. - - Returns: - The version token (``0.12.5``), or ``""``. - """ - try: - proc = _run(["uv", "--version"], cwd=directory) - except OSError: - return "" - words = str(proc.stdout or "").split() - if proc.returncode != 0 or len(words) < 2 or words[0] != "uv": - return "" - return words[1] - - def _check_call(argv: list[str], *, cwd: Path) -> list[str]: """Run a tool, surfacing a nonzero exit as :class:`ProjectError`. diff --git a/src/lightcone/engine/run.py b/src/lightcone/engine/run.py index cc4cbfb6..2b444875 100644 --- a/src/lightcone/engine/run.py +++ b/src/lightcone/engine/run.py @@ -14,16 +14,15 @@ from __future__ import annotations +import posixpath from collections.abc import Callable, Sequence from dataclasses import replace from pathlib import Path -from typing import Any from uuid import uuid4 -from lightcone.engine import container, sandbox +from lightcone.engine import container, plan, sandbox from lightcone.engine.execution_resources import TaskResources, worker_capacities from lightcone.engine.project import ( - SPEC_FILENAME, ProjectError, child_env, require_uv, @@ -50,7 +49,7 @@ def probe(project: Path, command: Sequence[str], *, cluster_id: str) -> sandbox. from lightcone.engine.compute.output import call, forwarding require_uv() - paths = input_paths(project, read_spec(project)) + paths = input_paths(project) with compute.connect(cluster_id) as client: resources = TaskResources().requirements( worker_capacities(client.scheduler_info()["workers"]), whole_worker=True, @@ -99,59 +98,39 @@ def _probe( return outcome -def read_spec(project: Path) -> dict[str, Any]: - """Read the project's spec, best-effort. +def input_paths(project: Path) -> list[Path]: + """Collect the files the analysis declares as inputs, best-effort. - A probe exists to debug a project, and a spec whose sub-analysis - references are stale is exactly when someone runs one. + Every analysis of the stack read in the working tree, not just the + project's own: a declared analysis has inputs of its own, and a probe + denied one is a denial the researcher cannot act on — the file is + declared, just not at the top of the tree. Bound rather than + validated, because a probe exists to debug a project, and a spec that + breaks a rule is exactly when someone runs one. Args: project: The project root. Returns: - The spec with sub-analyses merged in; the top-level document alone - if the tree will not resolve; an empty spec if there is none. + The resolved paths that exist; none where there is no analysis + ASTRA can read. """ - from astra.helpers import load_yaml, resolve_analysis_tree + from astra import api + from astra.cache import CacheError + from astra.sources.git import GitError - spec_path = project / SPEC_FILENAME - if not spec_path.exists(): - return {} - data: dict[str, Any] = load_yaml(spec_path) try: - return dict(resolve_analysis_tree(data, project)) - except Exception: - return data - - -def input_paths(project: Path, spec: dict[str, Any]) -> list[Path]: - """Collect the declared inputs that are filesystem paths. - - ASTRA's ``source`` is free-form — a URI, a dotted name, a path — so - the test for "is this a path" is whether it resolves to something that - exists. Anything else is somebody else's input kind. - - Args: - project: The project root, for resolving relative sources. - spec: The spec to read inputs from. - - Returns: - The resolved paths that exist. - """ - from astra.helpers import get_inputs - from astra.resolve import iter_analysis_nodes - + stack = api.bind(project).stack + except (api.NoAnalysisError, GitError, CacheError, OSError): + return [] + if stack is None: + return [] + above = posixpath.dirname(stack.analyses[stack.root].file) found: list[Path] = [] - # Every node, not just the root: a sub-analysis declares its own - # inputs, and a probe denied one is a denial the researcher cannot act - # on — the file is declared, just not at the top of the tree. - for _scope, node in iter_analysis_nodes(spec): - for declared in get_inputs(node): - source = declared.get("source") - if not isinstance(source, str) or not source: - continue - candidate = Path(source) - resolved = candidate if candidate.is_absolute() else project / candidate - if resolved.exists(): - found.append(resolved.resolve()) + for member, analysis in stack.determined(): + if member.identity.commit is None: + folder = project / plan.relative(member.identity.folder, above) + for declared in analysis.inputs.values(): + if (path := folder / declared.path).exists(): + found.append(path.resolve()) return list(dict.fromkeys(found)) diff --git a/src/lightcone/engine/sandbox/denial.py b/src/lightcone/engine/sandbox/denial.py index d317cdaf..c2069c73 100644 --- a/src/lightcone/engine/sandbox/denial.py +++ b/src/lightcone/engine/sandbox/denial.py @@ -164,11 +164,11 @@ def _render_data(path: Path) -> list[str]: return _message( f"cannot read {path}", [ - " if this is a data file, declare it as an input in astra.yaml:", + " if this is a data file, commit it in the project and declare it as an", + " input in astra.yaml, listed in the `inputs` of the output that reads it:", " inputs:", - " - id: ", - " type: data", - f" source: {path}", + " :", + f" path: data/{path.name}", ], ) diff --git a/src/lightcone/engine/templates/files/data-README.md.tmpl b/src/lightcone/engine/templates/files/data-README.md.tmpl index 608a07ae..591f53b2 100644 --- a/src/lightcone/engine/templates/files/data-README.md.tmpl +++ b/src/lightcone/engine/templates/files/data-README.md.tmpl @@ -4,7 +4,7 @@ Declared inputs live here, and `astra.yaml` references them by path: inputs: catalog: - source: data/catalog.fits + path: data/catalog.fits git-annex holds the bytes and git holds a pointer, so the repository records exactly which data produced a result without carrying its size. diff --git a/src/lightcone/engine/templates/files/gitattributes.tmpl b/src/lightcone/engine/templates/files/gitattributes.tmpl index ffd18568..6003dee3 100644 --- a/src/lightcone/engine/templates/files/gitattributes.tmpl +++ b/src/lightcone/engine/templates/files/gitattributes.tmpl @@ -5,11 +5,11 @@ # decides what counts as content: the default is nothing, and only outputs # and declared inputs opt out. Last matching line wins. # -# Manifests are exempted back out because they must be readable on a clone -# that has fetched no annex content. +# The `**/` line says the same for the results of an analysis in a folder +# below this one. * annex.largefiles=nothing * filter=annex results/** annex.largefiles=anything data/** annex.largefiles=anything .datalad/environments/*/image annex.largefiles=anything -results/**/.*.manifest.json annex.largefiles=nothing +**/results/** annex.largefiles=anything diff --git a/src/lightcone/engine/templates/files/results-README.md.tmpl b/src/lightcone/engine/templates/files/results-README.md.tmpl index c0925bc6..99f8c8e9 100644 --- a/src/lightcone/engine/templates/files/results-README.md.tmpl +++ b/src/lightcone/engine/templates/files/results-README.md.tmpl @@ -1,9 +1,10 @@ # results/ -Materialized outputs land here, one file per universe and output, named -from the output's id and the `format:` its spec declares: +Materialized results land here, one file each, at the place ASTRA gives +it from its output's id and the `format:` its spec declares: results//. -Beside each one is a `..manifest.json` recording how it was -made. +or `results/.` for an output built once. The record +of how each was made is at `.astra/records/`, at the same path with +`.json`. diff --git a/src/lightcone/engine/worker.py b/src/lightcone/engine/worker.py index c9ffa014..b6206e0a 100644 --- a/src/lightcone/engine/worker.py +++ b/src/lightcone/engine/worker.py @@ -1,13 +1,13 @@ -"""Making one output — the unit of work, and the only thing that runs a recipe. +"""Making one result — the unit of work, and the only thing that runs a recipe. Also an entry point: - python -m lightcone.engine.worker / + python -m lightcone.engine.worker which is what the ``[DATALAD RUNCMD]`` record in every materialization commit names, behind an engine-pinning ``uv run --no-project --with …`` — by version for a released engine, by source commit for a dev build. That -is why it is a module rather than an ``lc`` verb: it makes the output +is why it is a module rather than an ``lc`` verb: it makes the result unconditionally, commits nothing, and leaves the tree dirty by design — precisely the state ``lc materialize`` refuses to start from — so advertising it in ``lc --help`` would hand people a footgun. @@ -28,14 +28,15 @@ from __future__ import annotations import functools +import shutil import sys -from collections.abc import Callable, Mapping -from dataclasses import asdict, dataclass +from collections.abc import Callable +from dataclasses import dataclass from datetime import UTC, datetime from pathlib import Path -from typing import Literal +from typing import TYPE_CHECKING, Literal -from lightcone.engine import assets, container, dataset, identity, plan, project, sandbox +from lightcone.engine import assets, container, dataset, identity, plan, sandbox from lightcone.engine.execution_resources import TaskResources from lightcone.engine.plan import Key, Task from lightcone.engine.project import ( @@ -45,10 +46,8 @@ uv_prefix, ) -#: The commit a run is identified against: ``(sha, origin URL)``. Read once -#: by the driver and handed to every task, because the driver commits as -#: outputs land and HEAD therefore moves under the run. -Head = tuple[str, str] +if TYPE_CHECKING: + from astra.check import Status #: The shell a recipe's command is handed to. A recipe is a command line, #: not an argv — redirects and pipes are part of what people write — and @@ -63,10 +62,11 @@ class TaskResult: key: Key status: Literal["ok", "current", "behind", "failed", "blocked"] - #: The output's content identity. Present for the three states in - #: which the bytes on disk are what the spec asks for — this is what a - #: dependent compares against. - data_version: str = "" + #: The hash and size of the result's bytes, as a record gives them. + #: Present for the three states in which the bytes on disk are what + #: the spec asks for — this is what a dependent records having read. + content_hash: str = "" + size: int = 0 #: Why it did not finish. Shown to the user verbatim. reason: str = "" #: Console lines from the boundary: a downgrade notice, a denial. @@ -92,23 +92,16 @@ class RunContext: Each field is read or resolved exactly once, by whoever owns the run — the driver, or the rerun entry point — because a per-task read could answer differently mid-run: HEAD moves as the driver commits, - a runtime resolved twice could disagree, and a provenance field that - depends on task timing is worse than either answer. Frozen and - picklable, so it crosses to workers by value; one object, so the - next attestation field is one line here rather than an edit to five - signatures. + and a runtime resolved twice could disagree. Frozen and picklable, so + it crosses to workers by value. """ - #: The run's environment identity, checked either side of each recipe. - env_version: str - #: The run's ``(commit sha, origin URL)``. - head: Head - #: The run's content-hash memo for declared inputs. - versions: assets.Versions + #: The IRI of the run's environment, checked either side of each recipe. + environment: str + #: The commit the run started from, which every record it writes names. + commit: str #: The execution world — the host mechanism, or the project image. runtime: container.Runtime - #: The uv that converges environments this run. Attestation only. - uv_version: str # ============================================================================= @@ -121,6 +114,7 @@ def materialize( task: Task, context: RunContext, refresh: bool, + decided: Decided | None, foreign: dataset.LastWrite | None, *upstream: TaskResult, output: Callable[[str, bytes], None] | None = None, @@ -134,13 +128,16 @@ def materialize( Args: root: The project root. - task: The output to make. + task: The result to make. context: The run's driver-resolved facts. - refresh: Whether to remake an output that is merely behind. - foreign: The commit that last wrote the output's directory in - place of its own run record, or ``None`` — answered by the - driver, because history is git's and workers have no git; - handed to the one classification rule, where it is `stale`. + refresh: Whether to remake a result that is merely behind. + decided: What the driver found the result to be before the run, + or ``None`` where something upstream of it may be remade — + then only this task, once its upstreams have landed, can ask. + foreign: The commit that last wrote the result's files in place + of its own run, or ``None`` — answered by the driver, because + history is git's; handed to the one classification rule, + where it is `stale`. *upstream: The results of this task's dependencies, arriving as the futures it was given — which is what makes Dask the scheduler rather than a loop here. @@ -150,48 +147,80 @@ def materialize( What happened. Never raises. """ try: - return _materialize(root, task, context, refresh, foreign, upstream, output) + return _materialize(root, task, context, refresh, decided, foreign, upstream, output) except Exception as e: # the contract is that this function returns return TaskResult(task.key, "failed", reason=f"{type(e).__name__}: {e}") +@dataclass(frozen=True) +class Decided: + """What a result is, and the bytes its record gives if it is left alone.""" + + verdict: assets.Verdict + content_hash: str = "" + size: int = 0 + + +def decide( + found: tuple[Status, ...], environment: str, foreign: dataset.LastWrite | None +) -> Decided: + """Classify one result from what ASTRA's checker found at its places. + + Shared by the driver, which asks before a run, and a task downstream + of a remake, which asks again — so the two cannot word, or weigh, the + same finding differently. + + Args: + found: The result's statuses, one per place. + environment: The IRI of the run's environment. + foreign: The commit that last wrote the result's files, if not + its own run. + + Returns: + The verdict, with the recorded bytes of a result that has a + record. + """ + verdict = assets.classify(found=found, environment=environment, foreign=foreign) + if (record := found[0].record) is None: + return Decided(verdict) + return Decided(verdict, record.content_hash, record.size) + + def _materialize( root: Path, task: Task, context: RunContext, refresh: bool, + decided: Decided | None, foreign: dataset.LastWrite | None, upstream: tuple[TaskResult, ...], output: Callable[[str, bytes], None] | None, ) -> TaskResult: reported = {u.key: u for u in upstream if u.usable} if absent := [dep for dep in task.depends_on if dep not in reported]: - names = ", ".join(f"{u}/{o}" for u, o in absent) - return TaskResult(task.key, "blocked", reason=f"upstream did not finish: {names}") - - live = {key: u.data_version for key, u in reported.items()} - inputs = { - name: live[key] if (key := task.produced_by.get(name)) else context.versions.of(path) - for name, path in task.inputs.items() - } - manifest = assets.read(task.manifest_path) - verdict = assets.classify( - definition_version=task.definition_version, - env_version=context.env_version, - manifest=manifest, - inputs=inputs, - foreign=foreign, - ) - if verdict.calls_for_a_remake(refresh=refresh): - return execute(root, task, inputs, context, output=output) + return TaskResult( + task.key, "blocked", reason=f"upstream did not finish: {', '.join(absent)}" + ) + + if decided is None: + # Something upstream may have been remade, and only ASTRA can say + # whether it came out as the bytes this result recorded reading — + # a byte-identical rebuild stops the cascade here. + found = assets.check(root)[task.places[0].record] + decided = decide(found, context.environment, foreign) + if decided.verdict.calls_for_a_remake(refresh=refresh): + return execute(root, task, reported, context, output=output) # Left alone, so the bytes on disk stand. Their *recorded* digest, # never a recomputed one: on a clone that has fetched no annex content - # the files are dangling symlinks, and rehashing them would quietly - # report a different output. - assert manifest is not None and verdict.status != "stale" # the branch above + # the files are pointers, and rehashing them would quietly report a + # different output. return TaskResult( - task.key, verdict.status, data_version=manifest.data_version, reason=verdict.why + task.key, + decided.verdict.status, # type: ignore[arg-type] # stale was remade above + content_hash=decided.content_hash, + size=decided.size, + reason=decided.verdict.why, ) @@ -203,66 +232,93 @@ def _materialize( def execute( root: Path, task: Task, - input_versions: Mapping[str, str], + upstream: dict[Key, TaskResult], context: RunContext, *, output: Callable[[str, bytes], None] | None = None, ) -> TaskResult: """Run *task*'s recipe and record what it produced. - The output directory is reset first: the recipe owns it, and a file - left from a previous run would otherwise enter the content hash and be - committed as part of an output that never produced it. The context's - ``env_version`` is checked either side of the recipe, so a mid-run - lock edit cannot be recorded as if it had been in force. + What the result owns is removed first: a file left from a previous + run would otherwise be committed as part of a result that never + produced it. The context's environment is checked either side of the + recipe, so a mid-run lock edit cannot be recorded as if it had been + in force. Args: root: The project root. - task: The output to make. - input_versions: Each declared input's content identity, recorded - in the manifest as the chain. + task: The result to make. + upstream: The result of each task this one reads, by key. Their + hashes are what the record says this run read. context: The run's driver-resolved facts. output: Optional receiver forwarding recipe stdout and stderr bytes. Returns: - ``ok`` with the output's ``data_version``, or ``failed``. Commits - nothing and never touches git beyond reading HEAD. + ``ok`` with the hash and size of the result's bytes, or + ``failed``. Commits nothing and never touches git. """ + from astra import records + from astra.hashing import digest_file + resources = TaskResources.parse(task.resources) - if moved := _gate(root, context.env_version): + if moved := _gate(root, context.environment): return TaskResult(task.key, "failed", reason=moved) - # Never the parent directory: siblings share it now, and under Dask - # they are being written concurrently. What this output owns is its - # manifest and any file named after its id — the glob rather than the - # one declared path, so a spec that re-declares the output in another - # format leaves no stale payload behind. `.` needs a literal - # dot straight after the whole id, and an id cannot contain one, so it - # cannot reach a sibling, a longer id, another output's sidecar, or a - # scope directory of the same name. - task.output_path.parent.mkdir(parents=True, exist_ok=True) - read_paths = [p for p in task.inputs.values() if p.exists()] + # What the recipe is about to read, hashed before it runs: an input + # this clone does not hold is a refusal that should cost no compute. + read = [] + for entry, name in task.inputs.items(): + if (key := task.produced_by.get(entry)) is not None: + # What the upstream task reported, never a rehash: one left + # alone on a clone without its content is a pointer here. + read.append( + records.RecordedInput(entry, upstream[key].content_hash, upstream[key].size) + ) + else: + try: + assets.require_fetched(root / name) + hashed = digest_file(root / name) + except (OSError, ProjectError) as e: + return TaskResult( + task.key, "failed", reason=f"the input `{entry}` cannot be read: {e}" + ) + read.append(records.RecordedInput(entry, hashed.content_hash, hashed.size)) + + made = root / task.places[0].bytes + # Never the parent directory: siblings share it, and under Dask they + # are being written concurrently. What this result owns, at each of + # its places, is its record and any file named after it + # — the glob rather than the one declared path, so a spec that + # re-declares the output in another format leaves no stale payload + # behind. `.` cannot reach a sibling: ASTRA refuses an output + # that shares its id with a declaration, which is the only way one + # result's name could begin another's. + for place in task.places: + (root / place.bytes).parent.mkdir(parents=True, exist_ok=True) policy = container.policy_for( - context.runtime, read_paths, write_dir=task.output_path.parent, + context.runtime, + [path for name in task.inputs.values() if (path := root / name).exists()], + write_dir=made.parent, use_gpus=resources.gpus > 0, ) with sandbox.scope(policy): # Validate device visibility and container support before removing outputs. - task.manifest_path.unlink(missing_ok=True) - for stale in task.output_path.parent.glob(f"{task.output_id}.*"): - if stale.is_file() or stale.is_symlink(): - stale.unlink() - started_at = _now() + for place in task.places: + (root / place.record).unlink(missing_ok=True) + for stale in (root / place.bytes).parent.glob(f"{place.name}.*"): + if stale.is_file() or stale.is_symlink(): + stale.unlink() + started = datetime.now(UTC) outcome = sandbox.run( container.backend(context.runtime), policy, - [_SHELL, "-c", task.recipe], - cwd=root, + [_SHELL, "-c", task.command], + cwd=root / task.workdir, prefix=uv_prefix(root), env=child_env(), output=output, ) - finished_at = _now() + finished = datetime.now(UTC) if outcome.returncode != 0: return TaskResult( @@ -271,52 +327,40 @@ def execute( reason=f"the recipe exited {outcome.returncode}", notes=outcome.notes, ) - if moved := _gate(root, context.env_version): + if moved := _gate(root, context.environment): return TaskResult(task.key, "failed", reason=moved, notes=outcome.notes) # Guarded separately from the boundary catch above it, because these # two failures deserve different words: "your recipe failed" and "your # recipe worked and we could not record it" are different problems. - if not task.output_path.is_file(): - # `data_version` would answer for a directory — the `dir:` branch - # hashes it happily — so a recipe that ran `mkdir` on its output, - # or wrote some other name, would commit a well-formed digest of - # something that is not the output at all. - found = "a directory" if task.output_path.is_dir() else "nothing" + if not made.is_file(): + found = "a directory" if made.is_dir() else "nothing" return TaskResult( task.key, "failed", - reason=( - f"the recipe exited 0 but left {found} at " - f"{plan.declared_path(root, task.output_path)}" - ), + reason=f"the recipe exited 0 but left {found} at {task.places[0].bytes}", notes=outcome.notes, ) try: - sha, remote = context.head - data_version = assets.data_version(task.output_path) - assets.write( - task.manifest_path, - assets.Manifest( - output_id=task.output_id, - universe_id=task.universe_id, - recipe=task.recipe, - definition_version=task.definition_version, - env_version=context.env_version, - data_version=data_version, - decisions=dict(task.decisions), - input_versions=dict(input_versions), - git_sha=sha, - git_remote=remote, - lc_version=lc_version(), - uv_version=context.uv_version, - hermeticity=asdict(outcome.attestation), - started_at=started_at, - finished_at=finished_at, - image=context.runtime.manifest_image(), - ), - ) + digest = digest_file(made) + for place in task.places: + if place is not task.places[0]: + shutil.copyfile(made, root / place.bytes) + records.write_record( + root / place.record, + records.Record( + address=place.address, + content_hash=digest.content_hash, + size=digest.size, + commit=context.commit, + environment=context.environment, + runner=records.Runner("lightcone-cli", lc_version()), + inputs=tuple(read), + started=records.timestamp(started), + finished=records.timestamp(finished), + ), + ) except (OSError, ProjectError) as e: return TaskResult( task.key, @@ -324,22 +368,14 @@ def execute( reason=f"the recipe finished but its output could not be recorded: {e}", notes=outcome.notes, ) - return TaskResult(task.key, "ok", data_version=data_version, notes=outcome.notes) - - -def _now() -> str: - """The current instant, as a manifest timestamp. - - Milliseconds, not microseconds: RO-Crate consumers parse - ``schema:endTime`` with at most three fractional digits, and the - manifest is where that string is minted. - """ - return datetime.now(UTC).isoformat(timespec="milliseconds") + return TaskResult( + task.key, "ok", content_hash=digest.content_hash, size=digest.size, notes=outcome.notes + ) -def _gate(root: Path, env_version: str) -> str: +def _gate(root: Path, environment: str) -> str: """Empty while the environment is still the one the run started in.""" - if identity.env_version(root) == env_version: + if identity.environment(root) == environment: return "" return ( "the environment changed while the run was in flight — uv.lock, " @@ -353,7 +389,7 @@ def lc_version() -> str: """Report the running engine's version, empty for a bare source tree. The one lookup behind both places the engine attests itself: the - manifest's ``lc_version`` and the run record's version pin. One + record's ``runner`` and the run record's version pin. One function, so the two cannot disagree about which engine ran. Cached: the metadata scan walks ``sys.path``, both callers are once @@ -376,17 +412,17 @@ def lc_version() -> str: def main(argv: list[str]) -> int: """Run one task from the command line, unconditionally. - ``python -m lightcone.engine.worker /`` — what the + ``python -m lightcone.engine.worker `` — what the ``[DATALAD RUNCMD]`` record in every materialization commit names. A thin wrapper over :func:`execute`, so this is an entry point rather than a second implementation. Nothing is classified: a rerun is a rerun. The environment is converged here, not assumed: a rerun checks out the lock but never the ``.venv``, and the recipe's own ``uv run --no-sync`` - would silently hand it an *empty* environment while the manifest - recorded the lock's ``env_version``. The sync also carries - ``--locked``, so a lock that no longer matches ``pyproject.toml`` is a - loud refusal rather than a quiet relock. + would silently hand it an *empty* environment while the record named + the lock's. The sync also carries ``--locked``, so a lock that no + longer matches ``pyproject.toml`` is a loud refusal rather than a + quiet relock. The image, by contrast, is *found*, never built: the run record lists the committed archive in ``extra_inputs``, so ``datalad rerun`` has @@ -394,24 +430,23 @@ def main(argv: list[str]) -> int: commit, which nothing here makes. Args: - argv: One argument, ``/``. + argv: One argument, the task as lc names it (``fit#baseline``). Returns: 0 on success, 1 if the task failed, 2 on a bad argument or an unreadable project. """ - if len(argv) != 1 or "/" not in argv[0]: - print("usage: python -m lightcone.engine.worker /", file=sys.stderr) + if len(argv) != 1: + print("usage: python -m lightcone.engine.worker ", file=sys.stderr) return 2 - universe_id, _, output_id = argv[0].partition("/") try: root = declared_project() # The graph — and with it the task lookup — before any converge: # a typo'd target must cost nothing and mask nothing, and a # failing sync must not bury "no output `x`" under its own error. graph = plan.build(root) - if (task := graph.tasks.get((universe_id, output_id))) is None: + if (task := graph.tasks.get(argv[0])) is None: print(f"no output `{argv[0]}` in this project", file=sys.stderr) return 2 # This one-task run resolves its own runtime and HEAD, because it @@ -424,13 +459,11 @@ def main(argv: list[str]) -> int: result = execute( root, task, - _from_disk(task), + _from_disk(root, graph, task), RunContext( - env_version=identity.env_version(root), - head=dataset.head(root), - versions=assets.Versions(), + environment=identity.environment(root), + commit=dataset.head(root)[0], runtime=runtime, - uv_version=project.uv_version(root), ), ) except ProjectError as e: @@ -445,34 +478,22 @@ def main(argv: list[str]) -> int: return 0 -def _from_disk(task: Task) -> dict[str, str]: - """Upstream versions, read from the manifests already in the tree. +def _from_disk(root: Path, graph: plan.Graph, task: Task) -> dict[Key, TaskResult]: + """What *task* reads of other results, from the records already in the tree. - There is no graph in flight to take them from, and a single-task run - has nothing to share a memo with — the digests are read once each here - by construction. + There is no graph in flight to take them from, so each upstream's + record stands in for the result its task would have reported. """ - versions: dict[str, str] = {} - for name, path in task.inputs.items(): - if task.produced_by.get(name) is not None: - if (manifest := assets.read(assets.manifest_path(path))) is None: - raise ProjectError( - f"the input `{name}` has never been materialized — there is no " - f"manifest beside {path}. Run `lc materialize ` instead." - ) - versions[name] = manifest.data_version - else: - try: - versions[name] = assets.data_version(path) - except OSError as e: - # `data_version` reports an absent or unreadable path with - # the OS's own exception, and this is the entry point a - # `datalad rerun` lands on — so it names the declared input - # rather than unwinding a traceback at whoever reads that. - raise ProjectError( - f"the declared input `{name}` cannot be read: {e}" - ) from e - return versions + found = assets.check(root) if task.depends_on else {} + upstream = {} + for key in task.depends_on: + if (record := found[graph.tasks[key].places[0].record][0].record) is None: + raise ProjectError( + f"`{key}`, which `{task.key}` reads, has no valid record. " + "Run `lc materialize ` instead." + ) + upstream[key] = TaskResult(key, "current", record.content_hash, record.size) + return upstream if __name__ == "__main__": diff --git a/tests/conftest.py b/tests/conftest.py index 2dad8b87..f30151c8 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -5,6 +5,7 @@ import shutil import subprocess import textwrap +import tomllib from collections.abc import Callable, Iterable, Iterator from contextlib import contextmanager from pathlib import Path @@ -68,6 +69,13 @@ def ambient_uv(monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.delenv(name) +@pytest.fixture(autouse=True) +def astra_cache(tmp_path_factory: pytest.TempPathFactory, monkeypatch: pytest.MonkeyPatch) -> None: + """Keep ASTRA's cache — file hashes, fetched commits, papers — out of + the developer's own, so no test reads what another run left there.""" + monkeypatch.setenv("ASTRA_CACHE_DIR", str(tmp_path_factory.mktemp("astra-cache"))) + + @pytest.fixture(autouse=True) def machine_uv_config(monkeypatch: pytest.MonkeyPatch) -> None: """Blind the suite to the host's machine-level uv configuration. @@ -117,7 +125,7 @@ def fake_run(argv: list[str], *, cwd: Path) -> MagicMock: artifact = "uv.lock" if argv[1] == "lock" else ".venv" return MagicMock(returncode=0 if (project / artifact).exists() else 1) if argv[:2] == ["uv", "lock"]: - # Only has to exist and be non-empty — layer 1 parses no lock. + # Only has to exist and be non-empty — convergence parses no lock. (project / "uv.lock").write_text("version = 1\n") elif argv[:2] == ["uv", "sync"]: (project / ".venv" / "bin").mkdir(parents=True, exist_ok=True) @@ -248,13 +256,9 @@ def analysis(tmp_path: Path, real_tools: None) -> Callable[..., Path]: and `uv sync` together cost milliseconds. """ - def build( - spec: str, - files: dict[str, str] | None = None, - universes: dict[str, str] | None = None, - ) -> Path: + def build(spec: str, files: dict[str, str] | None = None) -> Path: root = tmp_path / "analysis" - for name in ("universes", "results", "data"): + for name in ("results", "data"): (root / name).mkdir(parents=True, exist_ok=True) (root / "pyproject.toml").write_text( @@ -265,8 +269,6 @@ def build( (root / ".gitattributes").write_text(templates.read("gitattributes.tmpl")) (root / ".gitignore").write_text(templates.read("gitignore.tmpl")) (root / "astra.yaml").write_text(textwrap.dedent(spec)) - for name, text in (universes or {"baseline": "id: baseline\ndecisions: {}\n"}).items(): - (root / "universes" / f"{name}.yaml").write_text(textwrap.dedent(text)) for name, text in (files or {}).items(): path = root / name path.parent.mkdir(parents=True, exist_ok=True) @@ -347,12 +349,29 @@ def engine_dist(tmp_path_factory: pytest.TempPathFactory) -> tuple[str, Path]: The wheel's exact version, and the directory serving it. """ dist = tmp_path_factory.mktemp("engine-dist") + repository = Path(__file__).parent.parent subprocess.run( ["uv", "build", "--wheel", "--out-dir", str(dist)], - cwd=Path(__file__).parent.parent, + cwd=repository, + check=True, + capture_output=True, + text=True, + ) + wheel = next(dist.glob("lightcone_cli-*.whl")) + # The engine's ASTRA pins are git sources, which a wheel's metadata does + # not carry — so the rerun's resolver is served those two as wheels too, + # built from the very commits `pyproject.toml` pins. + uv = tomllib.loads((repository / "pyproject.toml").read_text())["tool"]["uv"] + tools = uv["sources"]["astra-tools"] + pins = [ + f"astra-tools @ git+{tools['git']}@{tools['rev']}", + *uv["override-dependencies"], + ] + subprocess.run( + ["uv", "run", "--no-project", "--with", "pip", "--", "python", "-m", "pip", "wheel", + "--no-deps", "--wheel-dir", str(dist), *pins], check=True, capture_output=True, text=True, ) - wheel = next(dist.glob("*.whl")) return wheel.name.split("-")[1], dist diff --git a/tests/test_assets.py b/tests/test_assets.py index cde3d7d7..44804d09 100644 --- a/tests/test_assets.py +++ b/tests/test_assets.py @@ -1,11 +1,10 @@ -"""Tests for `lightcone.engine.assets` — an output's bytes, record, and -whether it is still current. +"""Tests for `lightcone.engine.assets` — whether a result is still current. -The classification table is the important part of this file. It is driven -the way both callers drive it — the worker with live content identities, -and `--check` with `None` for anything it has already decided will be -remade — because the entire justification for one rule is that those two -cannot disagree. +The classification table is the important part of this file. Whether a +record still describes the analysis is ASTRA's checker's to say, so the +rule is driven with the statuses that checker returns; what is pinned is +what lc makes of them, and the two things it adds — the environment a +result ran under, and whether its files were last written by its own run. The `stale` / `behind` split is the other half. `stale` means the artifact contradicts the project and must be remade; `behind` means it is still @@ -17,179 +16,85 @@ from __future__ import annotations -import hashlib -import json +from collections.abc import Callable from pathlib import Path import pytest - -from lightcone.engine import assets, dataset -from lightcone.engine.assets import Manifest, classify, data_version, output_path +from astra.check import Reason, Status +from astra.diagnostics import Location +from astra.records import Record, Runner +from astra.universes import Place +from conftest import CLUSTER_ID + +from lightcone.engine import assets, dataset, identity, materialize +from lightcone.engine.assets import classify from lightcone.engine.project import ProjectError - -def _manifest(**overrides: object) -> Manifest: - base: dict[str, object] = { - "output_id": "best_fit", - "universe_id": "baseline", - "recipe": "python src/fit.py {output}", - "definition_version": "sha256:code", - "env_version": "sha256:env", - "data_version": "sha256:data", - "decisions": {"method": "mcmc"}, - "input_versions": {"catalog": "sha256:cat"}, - "git_sha": "abc123", - "git_remote": "https://example/demo.git", - "lc_version": "0.4.2", - "uv_version": "0.12.5", - "hermeticity": {"mechanism": "landlock", "fs": "declared", "network": "allowed"}, - } - return Manifest(**{**base, **overrides}) # type: ignore[arg-type] - - -# ---- where an asset lives -------------------------------------------------- - - -def test_an_asset_is_addressed_by_its_path(tmp_path: Path) -> None: - """The path in a rendered recipe is the path on disk — no staging, no - scratch, no relocation.""" - assert output_path(tmp_path, "baseline", "best_fit", "csv") == ( - tmp_path / "results/baseline/best_fit.csv" - ) - - -# ---- content identity ------------------------------------------------------ - - -def test_the_same_bytes_hash_the_same(tmp_path: Path) -> None: - for name in ("a", "b"): - (tmp_path / name).mkdir() - (tmp_path / name / "fit.csv").write_text("x,y\n1,2\n") - assert data_version(tmp_path / "a") == data_version(tmp_path / "b") - - -def test_changed_bytes_move_it(tmp_path: Path) -> None: - (tmp_path / "fit.csv").write_text("x,y\n1,2\n") - before = data_version(tmp_path) - (tmp_path / "fit.csv").write_text("x,y\n1,3\n") - assert data_version(tmp_path) != before - - -def test_a_rename_moves_it(tmp_path: Path) -> None: - """The path goes into the hash beside the bytes: the same content under - a different name is a different output.""" - (tmp_path / "fit.csv").write_text("x,y\n") - before = data_version(tmp_path) - (tmp_path / "fit.csv").rename(tmp_path / "result.csv") - assert data_version(tmp_path) != before - - -def test_touching_a_file_does_not_move_it(tmp_path: Path) -> None: - """Content, never mtime. A file restored from history carries an old - timestamp and must not look changed because of it.""" - (tmp_path / "fit.csv").write_text("x,y\n") - before = data_version(tmp_path) - (tmp_path / "fit.csv").touch() - assert data_version(tmp_path) == before - - -def test_the_manifest_is_not_part_of_its_own_hash(tmp_path: Path) -> None: - """It carries the hash, so hashing it would be circular. Structural - now rather than an exclusion: the manifest is a sidecar beside the - output, so hashing one file cannot reach it.""" - output = tmp_path / "fit.csv" - output.write_text("x,y\n") - before = data_version(output) - assets.write(assets.manifest_path(output), _manifest()) - assert data_version(output) == before +_PLACE = Place( + "best_fit", + "best_fit#baseline", + ".astra/records/baseline/best_fit.json", + "results/baseline/best_fit.csv", +) -def test_a_file_and_a_directory_holding_it_are_different(tmp_path: Path) -> None: - """A declared input can be either, and the two must never collide. - Only the directory side is framed now — enough to keep them apart, - while a file's digest stays a plain sha256 of its own bytes.""" - (tmp_path / "one").mkdir() - (tmp_path / "one" / "fit.csv").write_text("x,y\n") - (tmp_path / "fit.csv").write_text("x,y\n") - assert data_version(tmp_path / "one") != data_version(tmp_path / "fit.csv") +# ---- where the record lives ------------------------------------------------ -@pytest.mark.parametrize("bad", ["", "/", "..", ".", "a/b", "results/../.."]) -def test_output_path_refuses_a_part_that_is_not_one_path_component( - tmp_path: Path, bad: str -) -> None: - """The path is composed, so a part carrying a separator would put an - output outside the tree every guard above it checked.""" - with pytest.raises(ProjectError): - assets.output_path(tmp_path, bad, "best_fit", "csv") - with pytest.raises(ProjectError): - assets.output_path(tmp_path, "baseline", bad, "csv") +def test_a_result_file_names_its_record() -> None: + assert assets.record_places("results/baseline/fit.json") == [ + ".astra/records/baseline/fit.json" + ] + assert assets.record_places("results/summary.csv") == [".astra/records/summary.json"] -@pytest.mark.parametrize("bad", ["", "/", "a/b", ".hidden"]) -def test_output_path_refuses_a_format_that_cannot_be_an_extension( - tmp_path: Path, bad: str -) -> None: - """A leading dot would make the output look like its own sidecar; a - separator would move it entirely.""" - with pytest.raises(ProjectError): - assets.output_path(tmp_path, "baseline", "best_fit", bad) +def test_a_result_of_an_analysis_in_a_folder_keeps_its_record_there() -> None: + assert assets.record_places("stage/results/v/mask.txt") == [ + "stage/.astra/records/v/mask.json" + ] -def test_output_path_refuses_an_id_carrying_a_dot(tmp_path: Path) -> None: - """The sidecar is the id with a leading dot and `.manifest.json` after - it, recovered by partitioning on the first dot — so an id carrying one - of its own would name a manifest for something else.""" - with pytest.raises(ProjectError, match="dot"): - assets.output_path(tmp_path, "baseline", "fit.plot", "png") +def test_a_dotted_file_name_offers_every_split() -> None: + """A name is ids joined by dots and a format may hold dots too, so the + file's name alone cannot say where one ends.""" + assert assets.record_places("results/u/stage.chain.tar.gz") == [ + ".astra/records/u/stage.json", + ".astra/records/u/stage.chain.json", + ".astra/records/u/stage.chain.tar.json", + ] -def test_the_manifest_is_named_from_the_id_never_the_format(tmp_path: Path) -> None: - """A format may contain dots, and `Path.stem` would answer `x.tar` for - `x.tar.gz`. Naming the sidecar from the id alone also keeps its path — - and so its history — across a re-declared serialization.""" - packed = assets.output_path(tmp_path, "baseline", "chain", "tar.gz") - plain = assets.output_path(tmp_path, "baseline", "chain", "npz") - assert assets.manifest_path(packed).name == ".chain.manifest.json" - assert assets.manifest_path(packed) == assets.manifest_path(plain) +@pytest.mark.parametrize( + "path", + [ + "data/catalog.csv", + "astra.yaml", + "results/a/b/fit.json", + "results/baseline/README", + ], +) +def test_a_file_at_no_results_place_has_no_record(path: str) -> None: + """Two levels under `results/` is no place ASTRA gives, and a name + with no format is nobody's bytes.""" + assert assets.record_places(path) == [] -def test_a_file_hashes_its_own_bytes_and_nothing_else(tmp_path: Path) -> None: - """Unframed, so the digest is one an outsider reproduces with - `sha256sum` — and the same number the crate publishes for that file.""" - output = tmp_path / "fit.csv" - body = b"x,y\n" - output.write_bytes(body) - assert data_version(output) == f"sha256:{hashlib.sha256(body).hexdigest()}" +# ---- content that is not there --------------------------------------------- -def test_an_unfetched_annexed_file_is_refused_not_hashed(tmp_path: Path) -> None: +def test_an_unfetched_annexed_file_is_refused(tmp_path: Path) -> None: """`filter=annex` leaves a pointer file where the content would be, so - the path *exists* and is readable. Hashing it would be a well-formed - answer to the wrong question, and would land in a manifest as if it - described the data.""" + the path *exists* and is readable — using it would be a well-formed + answer to the wrong question.""" pointer = tmp_path / "catalog.fits" - pointer.write_text( - "/annex/objects/SHA256E-s300000--4367c4a63392fa9b887bbcf046033d89.fits\n" - ) + pointer.write_text("/annex/objects/SHA256E-s300000--4367c4a63392fa9b887bbcf046033d89.fits\n") + assert assets.is_pointer(pointer) with pytest.raises(assets.ContentNotFetchedError, match="git annex get"): - data_version(pointer) + assets.require_fetched(pointer) -def test_a_directory_holding_an_unfetched_file_is_refused_too(tmp_path: Path) -> None: - """An output directory is hashed as a whole, so one absent file must - not be quietly folded in as its pointer.""" - (tmp_path / "fit.csv").write_text("a,b\n") - (tmp_path / "big.bin").write_text("/annex/objects/SHA256E-s9--abc.bin\n") - - with pytest.raises(assets.ContentNotFetchedError): - data_version(tmp_path) - - -def test_an_unfetched_locked_file_is_refused_like_an_unfetched_pointer( - tmp_path: Path, -) -> None: +def test_an_unfetched_locked_file_is_refused_like_an_unfetched_pointer(tmp_path: Path) -> None: """The other shape an annexed file takes. A researcher may run `git annex lock`, or set `annex.thin`, whenever they like — so detection cannot depend on which one lc's own writes happen to produce.""" @@ -198,115 +103,45 @@ def test_an_unfetched_locked_file_is_refused_like_an_unfetched_pointer( assert locked.is_symlink() and not locked.exists() with pytest.raises(assets.ContentNotFetchedError, match="git annex get"): - data_version(locked) + assets.require_fetched(locked) -def test_a_directory_holding_an_unfetched_locked_file_is_refused_too( - tmp_path: Path, -) -> None: - """The quiet one. A dangling symlink answers False to `is_file()`, so - filtering a directory walk on that alone drops the absent file from the - digest without a word — reporting a hash of the subset that happens to - be present, which is a worse lie than the pointer's.""" - (tmp_path / "fit.csv").write_text("a,b\n") - (tmp_path / "big.bin").symlink_to("../.git/annex/objects/xx/yy/SHA256E-s9--abc.bin") +def test_a_fetched_file_passes_in_either_shape(tmp_path: Path) -> None: + content = tmp_path / "object.fits" + content.write_text("x,y\n1,2\n") + locked = tmp_path / "catalog.fits" + locked.symlink_to(content) - with pytest.raises(assets.ContentNotFetchedError): - data_version(tmp_path) + assets.require_fetched(content) + assets.require_fetched(locked) -def test_a_broken_symlink_that_is_not_annexed_is_still_loud(tmp_path: Path) -> None: - """Not git-annex's doing, so not `git annex get`'s to fix — but it may - not vanish from the digest either.""" - (tmp_path / "fit.csv").write_text("a,b\n") - (tmp_path / "scratch.dat").symlink_to("/tmp/gone-with-the-scratch-dir") +def test_a_broken_symlink_that_is_not_annexed_is_not_called_unfetched(tmp_path: Path) -> None: + """Not git-annex's doing, so not `git annex get`'s to fix — whatever + opens the file reports it.""" + stray = tmp_path / "scratch.dat" + stray.symlink_to(tmp_path / "gone-with-the-scratch-dir") - with pytest.raises(FileNotFoundError): - data_version(tmp_path) + assets.require_fetched(stray) -def test_a_file_that_merely_mentions_the_prefix_is_still_hashed(tmp_path: Path) -> None: +def test_a_file_that_merely_mentions_the_prefix_is_content(tmp_path: Path) -> None: """The test is the prefix at the very start, as git-annex's own is — a script that talks about annex paths is not a pointer.""" script = tmp_path / "fit.py" script.write_text("# reads /annex/objects/ sometimes\nprint(1)\n") - assert data_version(script).startswith("sha256:") - - -def test_a_missing_path_is_an_error_not_an_empty_hash(tmp_path: Path) -> None: - """The failure mode worth refusing loudly: a constant digest for - everything absent would silently disable the whole chain.""" - with pytest.raises(FileNotFoundError): - data_version(tmp_path / "nothing") - - -def test_one_input_is_hashed_once_per_run(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - """A declared input is asked for once per `(universe, output)` that names - it — for a multiverse spec, the same bytes over and over. Eight - universes times four outputs sharing one catalog is thirty-two reads of - one file.""" - catalog = tmp_path / "catalog.txt" - catalog.write_text("measured\n") - hashed: list[Path] = [] - real = assets.data_version - monkeypatch.setattr(assets, "data_version", lambda p: (hashed.append(p), real(p))[1]) - - versions = assets.Versions() - digests = {versions.of(catalog) for _ in range(32)} - - assert len(hashed) == 1 - assert digests == {real(catalog)} - - -def test_the_memo_does_not_confuse_two_inputs(tmp_path: Path) -> None: - (tmp_path / "a.txt").write_text("one\n") - (tmp_path / "b.txt").write_text("two\n") - versions = assets.Versions() - - assert versions.of(tmp_path / "a.txt") != versions.of(tmp_path / "b.txt") - - -# ---- the manifest ---------------------------------------------------------- - - -def test_a_manifest_round_trips(tmp_path: Path) -> None: - written = _manifest() - sidecar = assets.manifest_path(tmp_path / "best_fit.csv") - assets.write(sidecar, written) - assert assets.read(sidecar) == written - + assert not assets.is_pointer(script) + assets.require_fetched(script) -def test_the_manifest_is_readable_json_with_the_schema_first(tmp_path: Path) -> None: - """It stays out of the annex precisely so a clone with no content - fetched can read it — including with a plain `grep`.""" - sidecar = assets.manifest_path(tmp_path / "best_fit.csv") - assets.write(sidecar, _manifest()) - text = sidecar.read_text() - assert next(iter(json.loads(text))) == "schema_version" - assert "best_fit" in text +def test_a_large_file_is_never_a_pointer(tmp_path: Path) -> None: + """git-annex's own bound: a pointer is small, so real content that + happens to begin like one is still content.""" + big = tmp_path / "big.bin" + big.write_bytes(b"/annex/objects/" + b"0" * (32 * 1024)) -def test_no_manifest_reads_as_none(tmp_path: Path) -> None: - assert assets.read(assets.manifest_path(tmp_path / "best_fit.csv")) is None - - -def test_an_unparseable_manifest_reads_as_none(tmp_path: Path) -> None: - """The safe direction: an unreadable record means make it again, not - trust it.""" - sidecar = assets.manifest_path(tmp_path / "best_fit.csv") - sidecar.write_text("{not json") - assert assets.read(sidecar) is None - - -def test_writing_a_manifest_replaces_the_previous_one_whole(tmp_path: Path) -> None: - sidecar = assets.manifest_path(tmp_path / "best_fit.csv") - assets.write(sidecar, _manifest()) - assets.write(sidecar, _manifest(data_version="sha256:second")) - - manifest = assets.read(sidecar) - assert manifest is not None and manifest.data_version == "sha256:second" - assert not list(tmp_path.glob("*.tmp")) + assert not assets.is_pointer(big) # ---- classification -------------------------------------------------------- @@ -314,90 +149,81 @@ def test_writing_a_manifest_replaces_the_previous_one_whole(tmp_path: Path) -> N # Every case names the environment it is classified against, because that # argument is what decides `behind`, and a default would hide it. -_ENV = "sha256:env" - +_ENV = "urn:lightcone:env:sha256:env" _TWEAK = dataset.LastWrite( "8d31f00" + "0" * 33, "tweak colors", "Ada", "ada@example.org", "2026-08-19" ) +_AT = Location(_PLACE.record, "", None, None) -def _classify(**overrides: object) -> assets.Verdict: - """Classify against the manifest `_manifest()` builds, unchanged.""" - call: dict[str, object] = { - "definition_version": "sha256:code", - "env_version": _ENV, - "manifest": _manifest(), - "inputs": {"catalog": "sha256:cat"}, - } - return classify(**{**call, **overrides}) # type: ignore[arg-type] +def _record(environment: str = _ENV) -> Record: + return Record( + _PLACE.address, "sha256:data", 8, "c" * 40, environment, Runner("lightcone-cli", "1.0") + ) -def test_a_never_materialized_output_is_stale() -> None: - verdict = _classify(manifest=None, inputs={}) - assert verdict.status == "stale" - assert verdict.calls_for_a_remake(refresh=False) - assert "never been materialized" in verdict.why + +def _unchanged(environment: str = _ENV, *, absent: bool = False) -> Status: + reasons = (Reason("ABSENT_RESULT", "warning", "the bytes are not here", _AT),) if absent else () + return Status( + _PLACE, "unchanged", "absent" if absent else "present", True, reasons, + _record(environment), (), + ) -def test_an_unchanged_output_is_current() -> None: - verdict = _classify() - assert verdict.status == "current" - assert not verdict.calls_for_a_remake(refresh=False) - assert verdict.why == "" +def _changed(*messages: str, environment: str = _ENV) -> Status: + reasons = tuple(Reason("STALE_RESULT", "error", message, _AT) for message in messages) + return Status(_PLACE, "stale", "present", True, reasons, _record(environment), ()) -def test_a_drifted_definition_is_stale() -> None: - """One reason covers the recipe and the decisions, because - `definition_version` is what both feed.""" - verdict = _classify(definition_version="sha256:other") - assert verdict.status == "stale" - assert "recipe or its decisions" in verdict.why +_MISSING = Status( + _PLACE, "missing", "absent", False, + (Reason("MISSING_RECORD", "error", "no record is at the place", _AT),), None, (), +) -def test_a_drifted_input_is_stale_and_says_which() -> None: - verdict = _classify(inputs={"catalog": "sha256:new"}) +def test_a_result_with_no_record_is_stale() -> None: + verdict = classify(found=[_MISSING], environment=_ENV) assert verdict.status == "stale" - assert "`catalog`" in verdict.why + assert verdict.calls_for_a_remake(refresh=False) + assert verdict.why == "no record is at the place" -def test_a_newly_declared_input_is_stale() -> None: - """The manifest has no version recorded for it, so the sets differ.""" - verdict = _classify(inputs={"catalog": "sha256:cat", "mask": "sha256:mask"}) - assert verdict.status == "stale" - assert "`mask`" in verdict.why +def test_an_unchanged_result_is_current() -> None: + verdict = classify(found=[_unchanged()], environment=_ENV) + assert verdict.status == "current" + assert not verdict.calls_for_a_remake(refresh=False) + assert not verdict.calls_for_a_remake(refresh=True) + assert verdict.why == "" -def test_an_input_the_output_no_longer_declares_is_stale() -> None: - """`definition_version` hashes the recipe and the decisions — neither of - which a dropped input moves — so nothing else would catch it and the - output would stay current with a dependency set that changed.""" - verdict = _classify(inputs={}) +def test_a_result_astra_finds_changed_is_stale_in_astras_words() -> None: + """Every reason, because a result can have moved in more than one way + and the user fixes them in one pass.""" + verdict = classify( + found=[_changed("the command changed", "`catalog` changed")], environment=_ENV + ) assert verdict.status == "stale" - assert "`catalog`" in verdict.why + assert verdict.why == "the command changed; `catalog` changed" -def test_an_input_that_will_be_remade_is_stale() -> None: - """`--check`'s sentinel. It cannot know whether a rebuild comes out - byte-identical, so `None` means "this is going to change" — deliberate - over-approximation, and the only difference between the two callers.""" - assert _classify(inputs={"catalog": None}).status == "stale" +def test_one_reason_found_at_two_places_is_said_once() -> None: + found = [_changed("the command changed"), _changed("the command changed")] + assert classify(found=found, environment=_ENV).why == "the command changed" -def test_the_sentinel_and_a_real_drift_give_the_same_verdict() -> None: - """The property that justifies one rule with two callers: for the same - state they agree, and `--check` differs from the worker only in what it - is able to know.""" - assert _classify(inputs={"catalog": None}) == _classify( - inputs={"catalog": "sha256:rebuilt"} - ) +def test_one_place_that_is_not_unchanged_makes_the_result_stale() -> None: + """The recipe writes every place in one run, so there is no remaking + half a result.""" + verdict = classify(found=[_unchanged(), _MISSING], environment=_ENV) + assert verdict.status == "stale" -def test_a_byte_identical_rebuild_stops_the_cascade() -> None: - """What content hashing buys, and what the sentinel deliberately gives - up: the worker sees the upstream came out the same and does not rerun, - where `--check` had to assume it would.""" - assert _classify(inputs={"catalog": "sha256:cat"}).status == "current" +def test_absent_bytes_alone_do_not_make_a_result_stale() -> None: + """A clone that has fetched no annex content holds results that are + perfectly up to date.""" + assert classify(found=[_unchanged(absent=True)], environment=_ENV).status == "current" # ---- behind, which is the whole point of the split ------------------------- @@ -407,59 +233,155 @@ def test_a_moved_environment_is_behind_and_not_stale() -> None: """The headline. One added dependency rewrites `uv.lock` for the whole project, and folding that into the rebuild trigger is what made a week-old result disappear over a plotting library.""" - verdict = _classify(env_version="sha256:moved") + verdict = classify(found=[_unchanged("urn:lightcone:env:sha256:old")], environment=_ENV) assert verdict.status == "behind" assert not verdict.calls_for_a_remake(refresh=False) + assert verdict.calls_for_a_remake(refresh=True) def test_the_reason_says_what_happened_and_not_where() -> None: - """The commit an output came from is a field of its manifest, not a - phrase in a sentence — a caller with a column for it reads the record. - Interpolating it here would also have to handle a repository with no - commit yet, which records an empty sha.""" - verdict = _classify(env_version="sha256:moved") + """The commit a result came from is a field of its record, not a + phrase in a sentence — a caller with a column for it reads the record.""" + verdict = classify(found=[_unchanged("urn:lightcone:env:sha256:old")], environment=_ENV) assert verdict.why == "made under an earlier environment" +def test_one_place_made_under_an_earlier_environment_is_behind() -> None: + found = [_unchanged(), _unchanged("urn:lightcone:env:sha256:old")] + assert classify(found=found, environment=_ENV).status == "behind" + + def test_stale_wins_over_behind() -> None: """Both moved. The artifact has to be remade either way, so the reason reported is the one that calls for the work — a `behind` verdict here would say "left alone" about something about to run.""" - verdict = _classify(definition_version="sha256:other", env_version="sha256:moved") + verdict = classify( + found=[_changed("the command changed", environment="urn:lightcone:env:sha256:old")], + environment=_ENV, + ) assert verdict.status == "stale" assert verdict.calls_for_a_remake(refresh=False) def test_a_foreign_write_is_stale_and_the_prose_is_composed_here() -> None: """History enters the one rule as a value — the caller with git hands - over the offending commit, and a hit is a contradiction: the manifest - no longer describes the bytes. The sentence, like every other why, - is this module's.""" - verdict = _classify(foreign=_TWEAK) + over the offending commit, and a hit is a contradiction: the record + no longer describes the bytes. ASTRA knows nothing of it, so the + sentence is this module's.""" + verdict = classify(found=[_unchanged()], environment=_ENV, foreign=_TWEAK) assert verdict.status == "stale" assert verdict.calls_for_a_remake(refresh=False) assert "8d31f00" in verdict.why and "tweak colors" in verdict.why assert "git show 8d31f00" in verdict.why -def test_a_definition_stale_output_keeps_its_own_why_over_a_foreign_write() -> None: - """Both call for the same remake, and the definition drift is the more - actionable reason to report.""" - verdict = _classify(definition_version="sha256:other", foreign=_TWEAK) +def test_a_result_astra_finds_stale_keeps_its_own_why_over_a_foreign_write() -> None: + """Both call for the same remake, and what changed in the analysis is + the more actionable reason to report.""" + verdict = classify(found=[_changed("the command changed")], environment=_ENV, foreign=_TWEAK) assert verdict.status == "stale" assert "tweak colors" not in verdict.why def test_a_foreign_write_wins_over_behind() -> None: - """A behind output is not wrong; a foreign-written one is — reporting + """A behind result is not wrong; a foreign-written one is — reporting "left alone" about something the run is about to remake is the one wrong answer, the same rule as stale-over-behind.""" - verdict = _classify(env_version="sha256:moved", foreign=_TWEAK) + verdict = classify( + found=[_unchanged("urn:lightcone:env:sha256:old")], environment=_ENV, foreign=_TWEAK + ) assert verdict.status == "stale" assert "tweak colors" in verdict.why -def test_an_environment_that_did_not_move_is_not_behind() -> None: - """The negative half of the sensitivity pair: `behind` has to be off by - default, or every output reports it forever and the signal is dead.""" - assert _classify(env_version=_ENV).status == "current" +# ---- asking ASTRA's checker about a real project --------------------------- + +_SPEC = """ +spec_version: "0.1.0" +inputs: + catalog: {path: data/catalog.txt} +decisions: + method: + options: + mcmc: {} + nested: {} +universes: + baseline: {decisions: {method: mcmc}} +outputs: + summary: + type: data + format: txt + inputs: [catalog] + recipe: + command: cp {inputs.catalog} {output} + fit: + type: data + format: txt + inputs: [summary] + decisions: [method] + recipe: + command: cp {inputs.summary} {output} +""" + +_SUMMARY = ".astra/records/summary.json" +_FIT = ".astra/records/baseline/fit.json" + + +def test_before_any_run_every_planned_result_is_missing_and_so_stale( + analysis: Callable[..., Path], +) -> None: + """Keyed by where the record would be, from the project root — the + one name the driver, the worker and `lc view` all hold for a result.""" + root = analysis(_SPEC, {"data/catalog.txt": "measured\n"}) + + found = assets.check(root) + + assert list(found) == [_SUMMARY, _FIT] + for statuses in found.values(): + assert [status.record_status for status in statuses] == ["missing"] + verdict = classify(found=statuses, environment=identity.environment(root)) + assert verdict.status == "stale" and verdict.why + + +def test_checking_writes_nothing(analysis: Callable[..., Path]) -> None: + root = analysis(_SPEC, {"data/catalog.txt": "measured\n"}) + assets.check(root) + assert dataset.status(root) == [] + + +def test_a_materialized_result_is_current_until_something_it_is_made_from_moves( + analysis: Callable[..., Path], inline: None +) -> None: + """The rule over the real checker, end to end: current after its own + run, behind against another environment, stale once its input changes + — and the result reading it stale with it.""" + root = analysis(_SPEC, {"data/catalog.txt": "measured\n"}) + assert materialize.materialize(root, [], cluster_id=CLUSTER_ID).ok + environment = identity.environment(root) + + found = assets.check(root) + for statuses in found.values(): + assert [status.record_status for status in statuses] == ["unchanged"] + assert statuses[0].record is not None + assert statuses[0].record.environment == environment + assert classify(found=statuses, environment=environment).status == "current" + assert classify(found=statuses, environment=environment + "0").status == "behind" + + (root / "data" / "catalog.txt").write_text("measured again\n") + dataset.save(root, [root / "data"], "new catalog") + + found = assets.check(root) + for record in (_SUMMARY, _FIT): + assert classify(found=found[record], environment=environment).status == "stale" + assert "catalog" in classify(found=found[_SUMMARY], environment=environment).why + + +def test_a_spec_astra_rejects_is_a_clean_error(analysis: Callable[..., Path]) -> None: + root = analysis(_SPEC.replace("inputs: [catalog]", "inputs: [missing]")) + with pytest.raises(ProjectError, match="does not validate"): + assets.check(root) + + +def test_no_spec_is_a_clean_error(tmp_path: Path) -> None: + with pytest.raises(ProjectError, match="no astra.yaml"): + assets.check(tmp_path) diff --git a/tests/test_cli.py b/tests/test_cli.py index 8fe25799..d5484987 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -220,7 +220,7 @@ def test_run_passes_the_command_through_untouched( @pytest.mark.parametrize("argv", [["run"], ["materialize"], ["run", "bad:name", "true"], - ["materialize", "baseline/first"]]) + ["materialize", "first#baseline"]]) def test_execution_requires_an_allocation_identity_before_project_discovery( runner: CliRunner, tmp_path: Path, monkeypatch: pytest.MonkeyPatch, argv: list[str], ) -> None: @@ -370,11 +370,11 @@ def test_targets_reach_the_engine( ) -> None: seen = _stub(monkeypatch) - result = runner.invoke(main, ["materialize", cluster, "baseline/fit", "report"]) + result = runner.invoke(main, ["materialize", cluster, "fit#baseline", "report"]) assert result.exit_code == 0, result.output assert seen == [ - ("materialize", (["baseline/fit", "report"], {"refresh": False, "cluster_id": cluster})) + ("materialize", (["fit#baseline", "report"], {"refresh": False, "cluster_id": cluster})) ] @@ -427,12 +427,12 @@ def test_check_exits_nonzero_when_something_would_run( answer off the exit status.""" from lightcone.engine.materialize import MaterializeReport - _stub(monkeypatch, check=MaterializeReport(planned={"baseline/fit": "no manifest"})) + _stub(monkeypatch, check=MaterializeReport(planned={"fit#baseline": "no record"})) result = runner.invoke(main, ["materialize", "--check"]) assert result.exit_code == 1 - assert "would run baseline/fit" in result.output + assert "would run fit#baseline" in result.output def test_check_treats_all_positionals_as_targets_without_querying_compute( @@ -446,9 +446,9 @@ def unexpected(*args: object, **kwargs: object) -> None: monkeypatch.setattr(compute, "connect", unexpected) monkeypatch.setattr(compute, "validate_id", unexpected) seen = _stub(monkeypatch) - result = runner.invoke(main, ["materialize", "--check", "baseline/first"]) + result = runner.invoke(main, ["materialize", "--check", "first#baseline"]) assert result.exit_code == 0 - assert seen == [("check", (["baseline/first"], {"refresh": False}))] + assert seen == [("check", (["first#baseline"], {"refresh": False}))] def test_check_explains_that_a_cluster_id_is_not_a_target(runner: CliRunner) -> None: @@ -488,14 +488,14 @@ def test_a_failure_exits_nonzero( _stub( monkeypatch, - materialize=MaterializeReport(failed=["baseline/fit"], blocked=["baseline/report"]), + materialize=MaterializeReport(failed=["fit#baseline"], blocked=["report#baseline"]), ) result = runner.invoke(main, ["materialize", CLUSTER_ID]) assert result.exit_code == 1 - assert "failed baseline/fit" in result.output - assert "blocked baseline/report" in result.output + assert "failed fit#baseline" in result.output + assert "blocked report#baseline" in result.output def test_engine_prose_is_rendered_as_written( @@ -521,14 +521,14 @@ def test_the_json_report_is_machine_readable( ) -> None: from lightcone.engine.materialize import MaterializeReport - _stub(monkeypatch, materialize=MaterializeReport(made=["baseline/fit"])) + _stub(monkeypatch, materialize=MaterializeReport(made=["fit#baseline"])) result = runner.invoke(main, ["materialize", CLUSTER_ID, "--json"]) assert json.loads(result.output) == { "ok": True, "up_to_date": False, - "made": ["baseline/fit"], + "made": ["fit#baseline"], "current": [], "behind": {}, "failed": [], @@ -575,15 +575,15 @@ def _report() -> object: return StatusReport( outputs=[ - OutputStatus("baseline/first", "current", "", "3f2a1c8ffff", "sha256:one"), + OutputStatus("first#baseline", "current", "", "3f2a1c8ffff", "sha256:one"), OutputStatus( - "baseline/second", + "second#baseline", "behind", "made under an earlier environment", "3f2a1c8ffff", "sha256:two", ), - OutputStatus("baseline/third", "stale", "the input `first` changed", "", ""), + OutputStatus("third#baseline", "stale", "the input `first` changed", "", ""), ] ) @@ -596,7 +596,7 @@ def test_status_shows_each_output_its_state_and_its_commit( result = runner.invoke(main, ["status"]) assert result.exit_code == 0 - for fragment in ("baseline/first", "current", "behind", "stale", "3f2a1c8"): + for fragment in ("first#baseline", "current", "behind", "stale", "3f2a1c8"): assert fragment in result.output @@ -667,13 +667,13 @@ def test_status_renders_a_foreign_write_as_a_stale_output( message = ( 'last changed by 8d31f00 ("tweak colors", Ada, 2026-08-19) rather ' - "than its run record, so the manifest no longer describes these " + "than its own run, so the record no longer describes these " "bytes — the next run remakes it; inspect first with `git show 8d31f00`" ) report = StatusReport( outputs=[ OutputStatus( - "baseline/first", + "first#baseline", "stale", message, "3f2a1c8ffff", @@ -717,7 +717,7 @@ def test_status_json_is_machine_readable( "counts": {"current": 1, "behind": 1, "stale": 1}, "outputs": [ { - "output": "baseline/first", + "output": "first#baseline", "status": "current", "why": "", "git_sha": "3f2a1c8ffff", @@ -725,7 +725,7 @@ def test_status_json_is_machine_readable( "foreign_write": "", }, { - "output": "baseline/second", + "output": "second#baseline", "status": "behind", "why": "made under an earlier environment", "git_sha": "3f2a1c8ffff", @@ -733,7 +733,7 @@ def test_status_json_is_machine_readable( "foreign_write": "", }, { - "output": "baseline/third", + "output": "third#baseline", "status": "stale", "why": "the input `first` changed", "git_sha": "", diff --git a/tests/test_composition.py b/tests/test_composition.py new file mode 100644 index 00000000..024eafae --- /dev/null +++ b/tests/test_composition.py @@ -0,0 +1,248 @@ +"""ASTRA 0.1's own shapes, run through lc end to end. + +One project modelled on the specification's worked example: an analysis +that declares another, drives its decision, pins results across +universes, imports one of the declared analysis's own universes, derives +a universe from another and switches an output on with `when`. What the +spec *means* is ASTRA's to test; these check that lc builds, places, +records and re-checks each kind of result it plans. +""" + +from __future__ import annotations + +import json +from collections.abc import Callable +from pathlib import Path + +import pytest +from conftest import CLUSTER_ID + +from lightcone.engine import dataset, materialize, plan + +ROOT = """ + spec_version: "0.1.0" + analyses: + stage: {path: stage} + inputs: + catalog: {path: data/catalog.csv} + decisions: + method: + options: {a: {}, b: {}} + extra: + when: [method.b] + options: {on: {}, off: {}} + outputs: + fit: + type: metric + format: txt + inputs: [catalog, stage.cleaned] + decisions: [method, stage.cut] + recipe: + command: >- + cat {inputs} > {output} && + echo {decisions.method} {decisions.stage.cut} >> {output} + diagnostic: + type: metric + format: txt + when: [method.b] + inputs: [fit] + decisions: [extra] + recipe: + command: "cp {inputs.fit} {output} && echo {decisions.extra} >> {output}" + comparison: + type: table + format: txt + inputs: ["fit#u1", "fit#u2", "stage.cleaned#stage_v1"] + recipe: + command: "cat {inputs} > {output}" + universes: + u1: + decisions: {method: b, extra: on, stage.cut: tight} + u2: + derived_from: u1 + decisions: {method: a} + stage_v1: {analysis: stage, universe: v1} +""" + +STAGE = """ + spec_version: "0.1.0" + inputs: + raw: {path: data/raw.csv} + decisions: + cut: + options: {tight: {}, loose: {}} + outputs: + cleaned: + type: data + format: csv + inputs: [raw] + decisions: [cut] + recipe: + command: "(cat {inputs.raw}; echo {decisions.cut}; basename $PWD) > {output}" + universes: + v1: + decisions: {cut: loose} +""" + +FILES = { + "stage/astra.yaml": STAGE, + "stage/data/raw.csv": "x\n1\n", + "data/catalog.csv": "a,b\n1,2\n", +} + +#: Every result the project plans, as lc names its task → where its bytes go. +PLACES = { + "stage.cleaned#u1": "results/u1/stage.cleaned.csv", + "fit#u1": "results/u1/fit.txt", + "diagnostic#u1": "results/u1/diagnostic.txt", + "stage.cleaned#u2": "results/u2/stage.cleaned.csv", + "fit#u2": "results/u2/fit.txt", + "stage/cleaned#v1": "stage/results/v1/cleaned.csv", + "comparison": "results/comparison.txt", +} + + +@pytest.fixture +def project(analysis: Callable[..., Path]) -> Path: + return analysis(ROOT, FILES) + + +@pytest.fixture +def built(project: Path, inline: None) -> Path: + report = materialize.materialize(project, [], cluster_id=CLUSTER_ID) + assert report.ok and sorted(report.made) == sorted(PLACES), report.warnings + return project + + +def test_each_planned_result_is_one_task_at_the_place_astra_gives_it(project: Path) -> None: + """Per universe, built once, driven, and belonging to the declared + analysis: four kinds of result, one naming rule. `diagnostic` is + planned only where its `when` holds, so `u2` has none.""" + graph = plan.build(project) + + assert {key: task.places[0].bytes for key, task in graph.tasks.items()} == PLACES + assert graph.tasks["stage/cleaned#v1"].places[0].record == ( + "stage/.astra/records/v1/cleaned.json" + ) + assert graph.tasks["comparison"].places[0].record == ".astra/records/comparison.json" + + +def test_a_pin_and_an_import_are_dependencies_like_any_other(project: Path) -> None: + """A pinned entry reads another universe's result, and an imported + universe reads the declared analysis's own — so the task built once + waits on all three, each under the entry exactly as written.""" + task = plan.build(project).tasks["comparison"] + + assert task.produced_by == { + "fit#u1": "fit#u1", + "fit#u2": "fit#u2", + "stage.cleaned#stage_v1": "stage/cleaned#v1", + } + assert task.inputs["stage.cleaned#stage_v1"] == "stage/results/v1/cleaned.csv" + + +def test_a_derived_universe_runs_under_its_effective_selection(project: Path) -> None: + """`u2` sets only `method`; `stage.cut` is inherited, and `extra`, + whose `when` no longer holds, is dropped with the output that needs it.""" + tasks = plan.build(project).tasks + + assert tasks["fit#u2"].decisions == {"method": "a", "stage.cut": "tight"} + assert tasks["diagnostic#u1"].decisions == {"extra": "on"} + assert "diagnostic#u2" not in tasks + + +def test_a_declared_analysis_s_recipe_runs_in_its_own_folder(built: Path) -> None: + """Driven or not, the recipe's working directory is the folder of the + analysis the output is declared in — while a driven result's bytes go + to the driver's `results/`, under the driver's selection.""" + driven = (built / PLACES["stage.cleaned#u1"]).read_text().split() + own = (built / PLACES["stage/cleaned#v1"]).read_text().split() + + assert driven[-2:] == ["tight", "stage"] + assert own[-2:] == ["loose", "stage"] + + +def test_every_result_is_recorded_committed_and_current(built: Path) -> None: + """The run ends as clean as it started, each result's record names it + by address and lists exactly its output's entries, and the checker — + asked again from scratch — finds nothing to do.""" + assert not dataset.status(built) + + record = json.loads((built / ".astra/records/comparison.json").read_text()) + assert record["address"] == "comparison" + assert [item["entry"] for item in record["generated_by"]["inputs"]] == [ + "fit#u1", + "fit#u2", + "stage.cleaned#stage_v1", + ] + owned = json.loads((built / "stage/.astra/records/v1/cleaned.json").read_text()) + assert owned["address"] == "cleaned#v1" + + status = materialize.status(built) + assert {o.output: o.status for o in status.outputs} == dict.fromkeys(PLACES, "current") + assert materialize.check(built, []).up_to_date + + +def test_an_input_of_the_declared_analysis_stales_what_reads_it(built: Path) -> None: + """The edit of the specification's own table: the declared analysis's + input changes, so everything made from it is stale in every universe + — and what never read it stays put.""" + (built / "stage/data/raw.csv").write_text("x\n2\n") + dataset.save(built, ["stage/data/raw.csv"], "new raw data") + + report = materialize.check(built, []) + + assert sorted(report.planned) == sorted(PLACES) + assert "content of `raw`" in report.planned["stage/cleaned#v1"] + + +def test_a_selection_edit_stales_only_the_universe_it_touches( + built: Path, inline: None +) -> None: + """`u2` now drives the declared analysis differently: its results are + remade, with the comparison that pins one of them, and `u1` is left + alone.""" + spec = built / "astra.yaml" + spec.write_text( + spec.read_text().replace( + "decisions: {method: a}", "decisions: {method: a, stage.cut: loose}" + ) + ) + dataset.save(built, ["astra.yaml"], "u2 loosens the cut") + + report = materialize.materialize(built, [], cluster_id=CLUSTER_ID) + + assert report.ok + assert sorted(report.made) == ["comparison", "fit#u2", "stage.cleaned#u2"] + assert (built / PLACES["stage.cleaned#u2"]).read_text().split()[-2] == "loose" + + +def test_a_target_is_a_task_or_a_name_in_every_universe(project: Path) -> None: + graph = plan.build(project) + + assert graph.resolve(["fit"]) == ["fit#u1", "fit#u2"] + assert graph.resolve(["stage/cleaned#v1"]) == ["stage/cleaned#v1"] + assert list(graph.closure(["fit#u2"]).tasks) == ["stage.cleaned#u2", "fit#u2"] + + +def test_an_analysis_declared_at_a_commit_is_refused_not_guessed( + analysis: Callable[..., Path], +) -> None: + """Reading results from a commit means reading them through their + records at that commit, which this engine does not do — so it says so + before anything runs.""" + project = analysis(ROOT, FILES) + head = dataset.head(project)[0] + spec = project / "astra.yaml" + spec.write_text( + spec.read_text() + .replace("stage: {path: stage}", f"stage: {{path: stage, commit: {head}}}") + .replace("extra: on, stage.cut: tight}", "extra: on}") + .replace('inputs: [catalog, stage.cleaned]', 'inputs: [catalog, "stage.cleaned#stage_v1"]') + .replace("decisions: [method, stage.cut]", "decisions: [method]") + .replace(" {decisions.stage.cut}", "") + ) + dataset.save(project, ["astra.yaml"], "pin stage to a commit") + + with pytest.raises(plan.ProjectError, match="lc does not build from"): + plan.build(project) diff --git a/tests/test_compute_output.py b/tests/test_compute_output.py index d1199406..420f467e 100644 --- a/tests/test_compute_output.py +++ b/tests/test_compute_output.py @@ -49,7 +49,7 @@ def detached_cluster(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Iterato def test_detached_probe_preserves_redirected_stdout_bytes( analysis: Callable[..., Path], detached_cluster: str, ) -> None: - root = analysis("version: '0.0.13'\nname: analysis\ninputs: []\noutputs: []\n") + root = analysis("spec_version: '0.1.0'\n") result = subprocess.run( [ sys.executable, "-c", "from lightcone.cli.commands import main; main()", @@ -69,7 +69,7 @@ def test_detached_probe_preserves_redirected_stdout_bytes( def test_probe_uses_allocation_environment_and_accepts_explicit_command_variables( analysis: Callable[..., Path], detached_cluster: str, monkeypatch: pytest.MonkeyPatch, ) -> None: - root = analysis("version: '0.0.13'\nname: analysis\ninputs: []\noutputs: []\n") + root = analysis("spec_version: '0.1.0'\n") name = f"LC_TEST_AFTER_LAUNCH_{uuid4().hex}" monkeypatch.setenv(name, "driver-only") command = ["python", "-c", f"import os; print(os.environ.get({name!r}, 'not-forwarded'))"] @@ -87,7 +87,7 @@ def test_probe_uses_allocation_environment_and_accepts_explicit_command_variable def test_interrupt_warns_that_the_remote_command_may_still_run( analysis: Callable[..., Path], detached_cluster: str, ) -> None: - root = analysis("version: '0.0.13'\nname: analysis\ninputs: []\noutputs: []\n") + root = analysis("spec_version: '0.1.0'\n") started = root / "results/started" process = subprocess.Popen( [ @@ -149,8 +149,8 @@ def test_failed_detached_recipe_forwards_diagnostics_without_corrupting_json( ) -> None: command = "printf 'recipe stdout\\r\\n'; printf 'recipe failure\\n' >&2; exit 19" root = analysis( - "version: '0.0.13'\nname: analysis\ninputs: []\noutputs:\n" - " - id: broken\n type: metric\n format: txt\n" + "spec_version: '0.1.0'\noutputs:\n" + " broken:\n type: metric\n format: txt\n" f" recipe:\n command: {json.dumps(command)}\n" ) result = subprocess.run( @@ -165,7 +165,7 @@ def test_failed_detached_recipe_forwards_diagnostics_without_corrupting_json( assert result.returncode == 1, result.stderr.decode(errors="replace") assert result.stdout, result.stderr.decode(errors="replace") report = json.loads(result.stdout) - assert report["failed"] == ["baseline/broken"] + assert report["failed"] == ["broken"] assert b"recipe stdout\r\n" in result.stderr assert b"recipe failure\n" in result.stderr diff --git a/tests/test_container.py b/tests/test_container.py index fd28ed6a..d4926588 100644 --- a/tests/test_container.py +++ b/tests/test_container.py @@ -253,7 +253,6 @@ def test_a_present_archive_is_loaded_once_and_reused( second = container.runtime_for_run(root, build=False) assert first.image_id == expected and second.image_id == expected - assert first.arch == "amd64" assert first.archive == f".datalad/environments/{image.tag(root)}/image" assert len(_argvs(fake, "podman", "load")) == 1 # the second run hit the store @@ -622,7 +621,8 @@ def test_a_foreign_arch_archive_refuses_before_loading( def test_the_matching_arch_loads(root: Path, fake: list[list[str]]) -> None: _write_archive(image.archive_path(root, image.tag(root))) - assert container.runtime_for_run(root, build=False).arch == "amd64" + container.runtime_for_run(root, build=False) + assert len(_argvs(fake, "podman", "load")) == 1 diff --git a/tests/test_container_smoke.py b/tests/test_container_smoke.py index 8d023438..5f5ecb95 100644 --- a/tests/test_container_smoke.py +++ b/tests/test_container_smoke.py @@ -31,16 +31,14 @@ REQUIRED_ENV = "LC_CONTAINER_TESTS_REQUIRED" _SPEC = """ -version: "0.0.13" -name: analysis +spec_version: "0.1.0" inputs: - - id: catalog - type: data - source: data/catalog.fits + catalog: + path: data/catalog.fits outputs: - - id: sums + sums: type: metric format: txt recipe: @@ -216,30 +214,24 @@ def test_the_probe_and_its_boundary(runtime: str, cproject: Path, cluster_id: st def test_materialize_end_to_end_in_the_image(runtime: str, cproject: Path, cluster_id: str) -> None: """The whole layer at once: the driver builds and commits the image, converges the in-image environment, runs the recipe behind the mount - table, records the runtime facts, and leaves the tree clean — through + table, and leaves the tree clean — through the real Dask cluster, not a stub.""" report = engine.materialize(cproject, [], cluster_id=cluster_id) assert report.ok, report.warnings - assert report.made == ["baseline/sums"] - assert (cproject / "results/baseline/sums.txt").read_text() == "4\n" + assert report.made == ["sums"] + assert (cproject / "results/sums.txt").read_text() == "4\n" assert not dataset.status(cproject) - manifest = assets.read(cproject / "results/baseline/.sums.manifest.json") - assert manifest is not None - assert manifest.hermeticity["mechanism"] == runtime - assert manifest.hermeticity["network"] == "allowed" - assert manifest.hermeticity["fs"] == "declared" - assert manifest.image is not None - assert manifest.image["id"] == _inspect_id(runtime, manifest.image["id"]) - assert manifest.image["archive"] == f".datalad/environments/{manifest.image['tag']}/image" + archive = image.archive_path(cproject, image.tag(cproject)) + assert archive.is_file() # The record lists the archive, so a rerun's `datalad get` fetches # the exact bytes before the worker runs. Parsed rather than matched # as text, because the record is datalad's format, not ours. record = dataset._git(["log", "-1", "--format=%B"], cwd=cproject) body = record.partition("=== Do not change lines below ===")[2].partition("^^^")[0] - assert json.loads(body)["extra_inputs"] == [manifest.image["archive"]] + assert json.loads(body)["extra_inputs"] == [archive.relative_to(cproject).as_posix()] def test_a_rerun_on_a_clone_fetches_the_archive_and_reproduces( @@ -264,8 +256,8 @@ def test_a_rerun_on_a_clone_fetches_the_archive_and_reproduces( monkeypatch.setattr(engine, "_engine_requirement", lambda: f"lightcone-cli=={version}") report = engine.materialize(cproject, [], cluster_id=cluster_id) assert report.ok, report.warnings - original = assets.read(cproject / "results/baseline/.sums.manifest.json") - assert original is not None + archive = image.archive_path(cproject, image.tag(cproject)).relative_to(cproject) + made = json.loads((cproject / ".astra/records/sums.json").read_text()) clone = tmp_path / "clone" dataset._git(["clone", "-q", str(cproject), str(clone)], cwd=tmp_path) @@ -274,7 +266,7 @@ def test_a_rerun_on_a_clone_fetches_the_archive_and_reproduces( dataset._git(["annex", "init", "-q", "clone"], cwd=clone) # No annex content: the archive here is a pointer until rerun gets it. with pytest.raises(assets.ContentNotFetchedError): - assets.require_fetched(clone / str(original.image["archive"])) # type: ignore[index] + assets.require_fetched(clone / archive) proc = subprocess.run( [sys.executable, "-c", "from datalad.api import rerun; rerun('HEAD')"], @@ -285,9 +277,8 @@ def test_a_rerun_on_a_clone_fetches_the_archive_and_reproduces( ) assert proc.returncode == 0, proc.stderr - rerun = assets.read(clone / "results/baseline/.sums.manifest.json") - assert rerun is not None - assert rerun.data_version == original.data_version + rerun = json.loads((clone / ".astra/records/sums.json").read_text()) + assert rerun["content_hash"] == made["content_hash"] assert not dataset.status(clone) diff --git a/tests/test_crate.py b/tests/test_crate.py index 4878b3b7..63932093 100644 --- a/tests/test_crate.py +++ b/tests/test_crate.py @@ -1,10 +1,11 @@ """Tests for `lightcone.engine.crate` — the publication view, rendered pure. -Everything here is disk-only: fixture manifests under ``tmp_path``, a +Everything here is disk-only: fixture records under ``tmp_path``, a hand-built graph, and a stub for "who last wrote this path" — no git, no -subprocess. Structure and ordering are asserted, never byte goldens; the -one byte-level claim is determinism, because convergence-by-materialize -rests on rendering twice at the same state yielding identical text. +subprocess. Structure and ordering are asserted, never +byte goldens; the one byte-level claim is determinism, because +convergence-by-materialize rests on rendering twice at the same state +yielding identical text. """ from __future__ import annotations @@ -15,12 +16,22 @@ from typing import Any import pytest +from astra import records +from astra.hashing import digest_file +from astra.universes import Place -from lightcone.engine import assets, crate +from lightcone.engine import crate from lightcone.engine.dataset import LastWrite from lightcone.engine.plan import Graph, Key, Task _DSID = "4b7b5c1e-0000-4000-8000-000000000000" +_RUN = "a" * 40 +_ENVIRONMENT = "urn:lightcone:env:sha256:" + "e" * 64 + +_REMOTE = "https://github.com/example/demo.git" + +#: What describes `first` in `baseline`. +_RECORD = ".astra/records/baseline/first.json" Writer = Callable[..., LastWrite] @@ -28,11 +39,10 @@ @pytest.fixture def project(tmp_path: Path) -> Path: root = tmp_path / "demo" - (root / "universes").mkdir(parents=True) - (root / "data").mkdir() - (root / "astra.yaml").write_text("name: demo\ndescription: A demo analysis.\n") - (root / "universes" / "baseline.yaml").write_text("id: baseline\n") - (root / "universes" / "alt.yaml").write_text("id: alt\n") + (root / "data").mkdir(parents=True) + (root / "astra.yaml").write_text( + 'spec_version: "0.1.0"\nlabel: Demo analysis\ndescription: A demo analysis.\n' + ) (root / "pyproject.toml").write_text('[project]\nname = "demo"\nlicense = "MIT"\n') (root / "uv.lock").write_text("version = 1\n") (root / ".python-version").write_text("3.12.5\n") @@ -40,70 +50,88 @@ def project(tmp_path: Path) -> Path: return root +def _place(name: str, universe: str = "", folder: str = "") -> Place: + """One result's place, laid out as ASTRA lays it out.""" + return Place( + name, + f"{name}#{universe}" if universe else name, + str(Path(folder, ".astra", "records", universe, f"{name}.json")), + str(Path(folder, "results", universe, f"{name}.txt")), + ) + + +def _task( + name: str, + universe: str = "", + *, + inputs: dict[str, str] | None = None, + produced_by: dict[str, Key] | None = None, + decisions: dict[str, str] | None = None, + workdir: str = "", +) -> Task: + place = _place(name, universe) + return Task( + place.address, + (place,), + f"make {name}", + workdir, + inputs or {}, + produced_by or {}, + {"method": "alpha"} if decisions is None else decisions, + ) + + def _made( root: Path, - universe_id: str, - output_id: str, + task: Task, *, - git_sha: str, - image: dict[str, str] | None = None, - inputs: dict[str, str] | None = None, - finished_at: str = "2026-08-19T10:05:00.000+00:00", -) -> Path: - output = root / "results" / universe_id / f"{output_id}.txt" + commit: str = _RUN, + finished: str = "2026-08-19T10:05:00.000Z", +) -> None: + """Leave what one run of *task* leaves: bytes and record.""" + place = task.places[0] + output = root / place.bytes output.parent.mkdir(parents=True, exist_ok=True) - output.write_text(f"{universe_id}/{output_id}\n") - assets.write( - assets.manifest_path(output), - assets.Manifest( - output_id=output_id, - universe_id=universe_id, - recipe=f"make {output_id}", - definition_version="sha256:def", - env_version="sha256:env", - data_version=f"sha256:{universe_id}-{output_id}", - decisions={"method": "alpha"}, - input_versions=inputs or {}, - git_sha=git_sha, - git_remote="https://github.com/example/demo.git", - lc_version="0.4.2", - hermeticity={"mechanism": "landlock"}, - started_at="2026-08-19T10:00:00.000+00:00", - finished_at=finished_at, - image=image, + output.write_text(f"{place.address}\n") + digest = digest_file(output) + records.write_record( + root / place.record, + records.Record( + address=place.address, + content_hash=digest.content_hash, + size=digest.size, + commit=commit, + environment=_ENVIRONMENT, + runner=records.Runner("lightcone-cli", "0.4.2"), + started="2026-08-19T10:00:00.000Z", + finished=finished, ), ) - return output -def _graph(root: Path, universes: tuple[str, ...] = ("baseline",)) -> Graph: +def _graph(universes: tuple[str, ...] = ("baseline",)) -> Graph: tasks: dict[Key, Task] = {} - for universe_id in universes: - first = root / "results" / universe_id / "first.txt" - tasks[(universe_id, "first")] = Task( - universe_id, - "first", - first, - "make first", - {"catalog": root / "data" / "catalog.csv"}, - {}, - {"method": "alpha"}, - "sha256:def", - ) - tasks[(universe_id, "second")] = Task( - universe_id, + for universe in universes: + first = _task("first", universe, inputs={"catalog": "data/catalog.csv"}) + second = _task( "second", - root / "results" / universe_id / "second.txt", - "make second", - {"first": first}, - {"first": (universe_id, "first")}, - {"method": "alpha"}, - "sha256:def", + universe, + inputs={"first": first.places[0].bytes}, + produced_by={"first": first.key}, ) + tasks |= {first.key: first, second.key: second} return Graph(tasks) -def _writer(*paths: Path) -> LastWrite: +def _materialized(root: Path, *keys: Key, universes: tuple[str, ...] = ("baseline",)) -> Graph: + """The two-step graph, with *keys* (every task when none) already made.""" + graph = _graph(universes) + for key in keys or graph.tasks: + _made(root, graph.tasks[key]) + return graph + + +def _writer(*paths: str | Path) -> LastWrite: return LastWrite("a" * 40, "irrelevant", "Ada Lovelace", "ada@example.org", "2026-08-19") @@ -112,9 +140,10 @@ def _render( graph: Graph, writer: Writer = _writer, keys: dict[str, str] | None = None, + remote: str = _REMOTE, ) -> dict[str, Any]: text = crate.render( - root, graph, license="MIT", dsid=_DSID, writer=writer, keys=keys or {} + root, graph, license="MIT", dsid=_DSID, remote=remote, writer=writer, keys=keys or {} ) loaded = json.loads(text) assert isinstance(loaded, dict) @@ -129,13 +158,15 @@ def _entities(document: dict[str, Any]) -> dict[str, dict[str, Any]]: def test_rendering_twice_at_the_same_state_is_byte_identical(project: Path) -> None: - _made(project, "baseline", "first", git_sha="aaa111") - _made(project, "baseline", "second", git_sha="aaa111") - graph = _graph(project) + graph = _materialized(project) keys = {"results/baseline/first.txt": "SHA256E-s24--" + "c" * 64 + ".txt"} - first = crate.render(project, graph, license="MIT", dsid=_DSID, writer=_writer, keys=keys) - second = crate.render(project, graph, license="MIT", dsid=_DSID, writer=_writer, keys=keys) + first, second = ( + crate.render( + project, graph, license="MIT", dsid=_DSID, remote=_REMOTE, writer=_writer, keys=keys + ) + for _ in range(2) + ) assert first == second @@ -144,11 +175,8 @@ def test_a_single_value_is_never_a_singleton_array(project: Path) -> None: """Compacted JSON-LD: one author, one parameter, one result are plain values — RO-Crate 1.1's recommendation, and what its validator checks. The arrays come from three builders, so this pins the serialization.""" - _made(project, "baseline", "first", git_sha="aaa111") - rendered = crate.render( - project, _graph(project), license="MIT", dsid=_DSID, writer=_writer, keys={} - ) - document = json.loads(rendered) + graph = _materialized(project, "first#baseline") + document = _render(project, graph) for entity in document["@graph"]: for name, value in entity.items(): assert not (isinstance(value, list) and len(value) == 1), (entity["@id"], name) @@ -159,28 +187,22 @@ def test_a_single_value_is_never_a_singleton_array(project: Path) -> None: def test_the_clock_never_enters_the_document(project: Path) -> None: """`datePublished` is the newest recorded instant — rocrate's own default stamps the current time, and this pins the override.""" - _made(project, "baseline", "first", git_sha="aaa111") - _made( - project, - "baseline", - "second", - git_sha="aaa111", - finished_at="2026-08-20T09:00:00.000+00:00", - ) + graph = _materialized(project, "first#baseline") + _made(project, graph.tasks["second#baseline"], finished="2026-08-20T09:00:00.000Z") - root_entity = _entities(_render(project, _graph(project)))["./"] + root_entity = _entities(_render(project, graph))["./"] - assert root_entity["datePublished"] == "2026-08-20T09:00:00.000+00:00" + assert root_entity["datePublished"] == "2026-08-20T09:00:00.000Z" # ---- the workflow-run context and the profiles ----------------------------- def test_the_context_carries_the_workflow_run_vocabulary(project: Path) -> None: - """Without it, `containerImage` and `sha256` are undefined terms that - JSON-LD silently drops on expansion — typed but inert.""" - _made(project, "baseline", "first", git_sha="aaa111") - document = _render(project, _graph(project)) + """Without it, `sha256` is an undefined term that JSON-LD silently + drops on expansion — typed but inert.""" + graph = _materialized(project, "first#baseline") + document = _render(project, graph) assert "https://w3id.org/ro/terms/workflow-run" in document["@context"] @@ -189,8 +211,8 @@ def test_the_crate_version_is_pinned_not_rocrates_default(project: Path) -> None """rocrate 0.16 moved its default to 1.3, whose context maps `ComputationalWorkflow` to a term the workflow profiles do not recognise — so the version is ours, and both halves must agree.""" - _made(project, "baseline", "first", git_sha="aaa111") - document = _render(project, _graph(project)) + graph = _materialized(project, "first#baseline") + document = _render(project, graph) entities = _entities(document) assert "https://w3id.org/ro/crate/1.1/context" in document["@context"] @@ -204,8 +226,8 @@ def test_the_crate_version_is_pinned_not_rocrates_default(project: Path) -> None def test_the_root_conforms_to_every_claimed_profile_and_declares_each( project: Path, ) -> None: - _made(project, "baseline", "first", git_sha="aaa111") - entities = _entities(_render(project, _graph(project))) + graph = _materialized(project, "first#baseline") + entities = _entities(_render(project, graph)) claimed = {ref["@id"] for ref in entities["./"]["conformsTo"]} assert "https://w3id.org/ro/wfrun/provenance/0.5" in claimed @@ -219,10 +241,8 @@ def test_the_root_conforms_to_every_claimed_profile_and_declares_each( def test_one_step_per_output_and_one_action_per_universe(project: Path) -> None: """A step is spec structure and an action is one execution — which is exactly how a multiverse maps onto the Provenance profile.""" - for universe_id in ("baseline", "alt"): - _made(project, universe_id, "first", git_sha="aaa111") - _made(project, universe_id, "second", git_sha="aaa111") - entities = _entities(_render(project, _graph(project, ("baseline", "alt")))) + graph = _materialized(project, universes=("baseline", "alt")) + entities = _entities(_render(project, graph)) steps = [e for e in entities.values() if e["@type"] == "HowToStep"] actions = [e for e in entities.values() if e["@type"] == "CreateAction"] @@ -235,78 +255,114 @@ def test_one_step_per_output_and_one_action_per_universe(project: Path) -> None: def test_outputs_sharing_a_commit_are_one_run(project: Path) -> None: - """The driver reads HEAD once and hands it down, so `git_sha` groups - manifests into runs — the whole Provenance layer's identity.""" - _made(project, "baseline", "first", git_sha="aaa111") - _made(project, "baseline", "second", git_sha="bbb222") - entities = _entities(_render(project, _graph(project))) + """The driver reads HEAD once and hands it down, so a record's + `commit` groups results into runs — the whole Provenance layer's + identity.""" + graph = _materialized(project, "first#baseline") + _made(project, graph.tasks["second#baseline"], commit="b" * 40) + entities = _entities(_render(project, graph)) organizers = {i: e for i, e in entities.items() if e["@type"] == "OrganizeAction"} - assert set(organizers) == {"#organize-aaa111", "#organize-bbb222"} + assert set(organizers) == {"#organize-aaaaaaa", "#organize-bbbbbbb"} for organizer in organizers.values(): run = entities[organizer["result"]["@id"]] assert run["@type"] == "CreateAction" assert run["instrument"]["@id"] == "astra.yaml" + assert organizers["#organize-aaaaaaa"]["object"] == {"@id": "#control-baseline-first"} + assert entities["#control-baseline-first"]["instrument"] == {"@id": "#step-first"} + + +def test_a_run_is_timed_and_attributed_from_its_records(project: Path) -> None: + """Start, end and the engine are the ones the run recorded — never + the clock's or the installed engine's, or two collaborators would + re-commit the crate at each other forever.""" + graph = _materialized(project, "first#baseline") + _made(project, graph.tasks["second#baseline"], finished="2026-08-20T09:00:00.000Z") + entities = _entities(_render(project, graph)) + + organizer = entities["#organize-aaaaaaa"] + engine = entities[organizer["instrument"]["@id"]] + assert engine["softwareVersion"] == "0.4.2" + assert engine["@id"] == "https://pypi.org/project/lightcone-cli/0.4.2/" + run = entities[organizer["result"]["@id"]] + # A numeric offset, which is the form the Run Crate shapes read. + assert run["startTime"] == "2026-08-19T10:00:00.000+00:00" + assert run["endTime"] == "2026-08-20T09:00:00.000+00:00" + assert _RUN in run["description"] def test_an_action_chains_its_inputs_and_its_environment(project: Path) -> None: - _made(project, "baseline", "first", git_sha="aaa111", inputs={"catalog": "sha256:cafe"}) - _made(project, "baseline", "second", git_sha="aaa111") - entities = _entities(_render(project, _graph(project))) + graph = _materialized(project) + entities = _entities(_render(project, graph)) - actions = { - e["name"]: e for e in entities.values() if e["@type"] == "CreateAction" - } + actions = {e["name"]: e for e in entities.values() if e["@type"] == "CreateAction"} first = actions["run of `first` in universe `baseline`"] second = actions["run of `second` in universe `baseline`"] first_objects = {ref["@id"] for ref in first["object"]} assert {"uv.lock", ".python-version", "pyproject.toml", "data/catalog.csv"} <= first_objects assert "results/baseline/first.txt" in {ref["@id"] for ref in second["object"]} assert second["result"] == {"@id": "results/baseline/second.txt"} - assert second["description"] == "make second" - assert entities["results/baseline/second.txt"]["version"] == "sha256:baseline-second" + assert second["description"] == graph.tasks["second#baseline"].command + made = digest_file(project / "results/baseline/second.txt") + assert entities["results/baseline/second.txt"]["version"] == made.content_hash -def test_the_manifest_is_in_the_crate_and_about_its_dataset(project: Path) -> None: - _made(project, "baseline", "first", git_sha="aaa111") - entities = _entities(_render(project, _graph(project))) +def test_the_record_is_in_the_crate_and_about_its_result(project: Path) -> None: + graph = _materialized(project, "first#baseline") + entities = _entities(_render(project, graph)) - manifest = entities["results/baseline/.first.manifest.json"] - assert manifest["about"] == {"@id": "results/baseline/first.txt"} - assert entities["results/baseline/first.txt"]["subjectOf"] == { - "@id": "results/baseline/.first.manifest.json" - } + assert entities[_RECORD]["about"] == {"@id": "results/baseline/first.txt"} + assert entities[_RECORD]["encodingFormat"] == "application/json" + result = entities["results/baseline/first.txt"] + assert result["name"] == "first#baseline" + assert result["subjectOf"] == {"@id": _RECORD} -def test_a_containerized_output_names_its_committed_archive(project: Path) -> None: - """Identity and payload as one entity: the archive file in the crate, - carrying the config-blob id the execution pinned.""" - _made( - project, - "baseline", - "first", - git_sha="aaa111", - image={ - "tag": "lc-env-x", - "id": "sha256:beef", - "archive": ".datalad/environments/lc-env-x/image", - "arch": "amd64", - }, - ) - entities = _entities(_render(project, _graph(project))) +def test_a_result_without_a_valid_record_is_not_described_as_made(project: Path) -> None: + """Bytes alone are not a run: the record is what says + which commit, when and under what, so without one that validates + there is no action to state.""" + graph = _materialized(project, "first#baseline") + (project / ".astra/records/baseline/first.json").write_text('{"address": "first#baseline"}\n') + entities = _entities(_render(project, graph)) - archive = entities[".datalad/environments/lc-env-x/image"] - assert set(archive["@type"]) == {"File", "ContainerImage"} - assert archive["sha256"] == "beef" - action = next(e for e in entities.values() if e["@type"] == "CreateAction") - assert action["containerImage"] == {"@id": ".datalad/environments/lc-env-x/image"} + assert "results/baseline/first.txt" not in entities + assert not [e for e in entities.values() if e["@type"] == "CreateAction"] + + +def test_a_result_built_once_has_no_universe_in_its_ids(project: Path) -> None: + task = _task("summary", decisions={}) + _made(project, task) + entities = _entities(_render(project, Graph({task.key: task}))) + + result = entities["results/summary.txt"] + assert result["name"] == "summary" + assert result["subjectOf"] == {"@id": ".astra/records/summary.json"} + control = entities["#control-summary"] + assert control["instrument"] == {"@id": "#step-summary"} + assert entities[control["object"]["@id"]]["name"] == "run of `summary`" + + +def test_a_declared_analysis_that_runs_recipes_brings_its_spec(project: Path) -> None: + """A driven analysis's result belongs to the project, under the name + the project gives it — and the spec holding its recipe is part of + what the deposit must carry.""" + (project / "stage").mkdir() + (project / "stage" / "astra.yaml").write_text('spec_version: "0.1.0"\n') + task = _task("stage.out", "baseline", workdir="stage") + _made(project, task) + entities = _entities(_render(project, Graph({task.key: task}))) + + assert entities["stage/astra.yaml"]["encodingFormat"] == "application/yaml" + assert entities["results/baseline/stage.out.txt"]["name"] == "stage.out#baseline" + assert entities["#control-baseline-stage.out"]["instrument"] == {"@id": "#step-stage.out"} def test_the_person_is_the_saving_commits_author(project: Path) -> None: - """The manifest's `git_sha` is the commit the run *started* at — the + """The record's `commit` is the one the run *started* at — the author comes from the commit that saved the output, via the writer.""" - _made(project, "baseline", "first", git_sha="aaa111") - entities = _entities(_render(project, _graph(project))) + graph = _materialized(project, "first#baseline") + entities = _entities(_render(project, graph)) person = entities["mailto:ada@example.org"] assert person["name"] == "Ada Lovelace" @@ -316,8 +372,8 @@ def test_the_person_is_the_saving_commits_author(project: Path) -> None: def test_decision_values_point_back_at_their_parameter(project: Path) -> None: - _made(project, "baseline", "first", git_sha="aaa111") - entities = _entities(_render(project, _graph(project))) + graph = _materialized(project, "first#baseline") + entities = _entities(_render(project, graph)) value = entities["#value-baseline-first-method"] assert value["value"] == "alpha" @@ -328,12 +384,15 @@ def test_decision_values_point_back_at_their_parameter(project: Path) -> None: def test_a_never_materialized_project_still_describes_its_workflow(project: Path) -> None: """No outputs, no runs — the crate is the workflow and the environment, - dated by the spec's own last commit rather than the clock.""" - entities = _entities(_render(project, _graph(project))) + dated by the spec's own last commit rather than the clock, and titled + by the spec's `label`.""" + entities = _entities(_render(project, _graph())) assert entities["./"]["datePublished"] == "2026-08-19" assert not [e for e in entities.values() if e["@type"] == "CreateAction"] - assert "astra.yaml" in entities + assert entities["astra.yaml"]["name"] == "Demo analysis" + assert entities["./"]["name"] == "Demo analysis" + assert entities["./"]["description"] == "A demo analysis." def test_the_license_is_a_local_entity_never_a_minted_url(project: Path) -> None: @@ -341,7 +400,7 @@ def test_the_license_is_a_local_entity_never_a_minted_url(project: Path) -> None spdx.org URL could be a fabricated dead link — a local CreativeWork carries the declared string instead, and a URL license is used as given.""" - entities = _entities(_render(project, _graph(project))) + entities = _entities(_render(project, _graph())) assert entities["./"]["license"] == {"@id": "#license"} assert entities["#license"] == { @@ -351,6 +410,23 @@ def test_the_license_is_a_local_entity_never_a_minted_url(project: Path) -> None } +def test_the_remote_is_where_the_workflow_and_its_tools_are_found(project: Path) -> None: + """A recipe's tool is identified under the repository's own URL when + there is one, and by a minted `urn:uuid` — never a guessed link — + when there is none.""" + graph = _materialized(project, "first#baseline") + + entities = _entities(_render(project, graph)) + assert entities["astra.yaml"]["url"] == _REMOTE + assert entities[f"{_REMOTE}#tool/first"]["url"] == _REMOTE + + entities = _entities(_render(project, graph, remote="")) + assert "url" not in entities["astra.yaml"] + tool = entities["#step-first"]["workExample"]["@id"] + assert tool.startswith("urn:uuid:") + assert "url" not in entities[tool] + + # ---- per-file integrity ---------------------------------------------------- @@ -358,10 +434,10 @@ def test_an_output_carries_the_checksum_from_its_annex_key(project: Path) -> Non """The keys are repository state, so a bytes-free clone renders the same claims — and the hex in a SHA256E key is the raw sha256 an archive can verify with `sha256sum` after a `git archive` deposit.""" - _made(project, "baseline", "first", git_sha="aaa111") + graph = _materialized(project, "first#baseline") digest = "d" * 64 keys = {"results/baseline/first.txt": f"SHA256E-s21--{digest}.txt"} - entities = _entities(_render(project, _graph(project), keys=keys)) + entities = _entities(_render(project, graph, keys=keys)) output = entities["results/baseline/first.txt"] assert output["@type"] == "File" @@ -370,22 +446,22 @@ def test_an_output_carries_the_checksum_from_its_annex_key(project: Path) -> Non def test_an_output_is_one_file_not_a_dataset_of_parts(project: Path) -> None: - """One output is one file, so there is nothing to enumerate — and the - manifest hangs off it as `subjectOf` rather than as a part of it.""" - _made(project, "baseline", "first", git_sha="aaa111") - entities = _entities(_render(project, _graph(project))) + """One output is one file, so there is nothing to enumerate — and its + record hangs off it as `subjectOf` rather than as a part of it.""" + graph = _materialized(project, "first#baseline") + entities = _entities(_render(project, graph)) output = entities["results/baseline/first.txt"] assert "hasPart" not in output - assert output["subjectOf"] == {"@id": "results/baseline/.first.manifest.json"} + assert output["subjectOf"] == {"@id": _RECORD} def test_a_non_sha256_key_yields_size_and_no_digest(project: Path) -> None: """`annex.backend` is the researcher's to set, and a wrong checksum is worse than none — the publish-neither discipline.""" - _made(project, "baseline", "first", git_sha="aaa111") + graph = _materialized(project, "first#baseline") keys = {"results/baseline/first.txt": "MD5E-s21--" + "e" * 32 + ".txt"} - entities = _entities(_render(project, _graph(project), keys=keys)) + entities = _entities(_render(project, graph, keys=keys)) part = entities["results/baseline/first.txt"] assert part["contentSize"] == "21" @@ -397,11 +473,11 @@ def test_pointer_shaped_bytes_never_become_a_checksum(project: Path) -> None: git-annex cannot answer at all, and an unlocked pointer file reads perfectly well — so the byte fallback re-checks the pointer shape instead of publishing a well-formed digest of the pointer text.""" - _made(project, "baseline", "first", git_sha="aaa111") + graph = _materialized(project, "first#baseline") pointer = "/annex/objects/SHA256E-s300--" + "a" * 64 + ".csv\n" (project / "data" / "catalog.csv").write_text(pointer) - entities = _entities(_render(project, _graph(project), keys={})) + entities = _entities(_render(project, graph, keys={})) catalog = entities["data/catalog.csv"] assert "sha256" not in catalog @@ -414,40 +490,13 @@ def test_git_carried_files_are_hashed_by_their_bytes(project: Path) -> None: pure.""" import hashlib - _made(project, "baseline", "first", git_sha="aaa111") - entities = _entities(_render(project, _graph(project))) + graph = _materialized(project, "first#baseline") + entities = _entities(_render(project, graph)) body = (project / "uv.lock").read_bytes() assert entities["uv.lock"]["sha256"] == hashlib.sha256(body).hexdigest() assert entities["uv.lock"]["contentSize"] == str(len(body)) -def test_an_out_of_tree_input_publishes_no_checksum( - project: Path, tmp_path: Path -) -> None: - """Its recorded input_versions digest is lc's *framed* hash, not a - raw sha256 — publishing it under the workflow-run term would be a - checksum nothing can verify. The manifests keep the full story.""" - catalog = tmp_path / "shared" / "catalog.csv" - catalog.parent.mkdir() - catalog.write_text("a,b\n") - task = Task( - "baseline", - "first", - project / "results/baseline/first.txt", - "make first", - {"catalog": catalog}, - {}, - {}, - "sha256:def", - ) - _made(project, "baseline", "first", git_sha="aaa111", inputs={"catalog": "sha256:cafe"}) - entities = _entities(_render(project, Graph({("baseline", "first"): task}))) - - external = entities[catalog.as_uri()] - assert "sha256" not in external - assert "contentSize" not in external - - def _as_list(value: Any) -> list[Any]: return value if isinstance(value, list) else [value] diff --git a/tests/test_crate_smoke.py b/tests/test_crate_smoke.py index 067dc3b2..7df6b5ff 100644 --- a/tests/test_crate_smoke.py +++ b/tests/test_crate_smoke.py @@ -9,8 +9,7 @@ At RECOMMENDED there is a known floor, pinned as a *set* rather than a count: the workflow's id is the file in the crate and cannot be an http -URI, the image lives in the annex and has no registry, and lc knows no -publisher and no author affiliation. A new failure is a regression; a +URI, and lc knows no publisher and no author affiliation. A new failure is a regression; a disappearing one is the floor to shrink. """ @@ -38,8 +37,6 @@ _FLOOR = { # the workflow's id is its path in the crate, and the shape wants http "process-run-crate-0.5_5.1", - # the image is bytes in the annex; there is no registry to name - "process-run-crate-0.5_13.2", # lc knows no publishing organization and no author affiliation "ro-crate-1.1_22.3", "ro-crate-1.1_30.2", @@ -47,37 +44,41 @@ } _SPEC = """ -version: "0.0.13" -name: analysis +spec_version: "0.1.0" +label: analysis description: A two-step analysis for the crate smoke test. inputs: - - id: catalog - type: data - source: data/catalog.fits + catalog: + path: data/catalog.fits + +decisions: + method: + label: Method + options: + alpha: {label: alpha} + beta: {label: beta} outputs: - - id: first + first: type: metric format: txt + inputs: [catalog] decisions: [method] recipe: command: echo {decisions.method} > {output} - - id: second + second: type: report format: txt inputs: [first] recipe: command: cat {inputs.first} > {output} -decisions: - method: - label: Method - default: alpha - options: - alpha: {label: alpha} - beta: {label: beta} +universes: + baseline: + decisions: + method: alpha """ @@ -117,11 +118,10 @@ def test_a_materialized_crate_validates_against_the_provenance_profile( root = analysis( _SPEC, files={"data/catalog.fits": "stars\n", "README.md": "# analysis\n\nA demo.\n"}, - universes={"baseline": "id: baseline\ndecisions:\n method: alpha\n"}, ) pyproject = root / "pyproject.toml" pyproject.write_text(pyproject.read_text() + 'license = "MIT"\n') - # A remote gives the tools http ids; the manifests record origin. + # A remote gives the tools http ids. dataset._git(["remote", "add", "origin", "https://github.com/example/analysis.git"], cwd=root) dataset.save(root, [root], "license and remote") diff --git a/tests/test_dataset.py b/tests/test_dataset.py index 8d50a93c..b9a2f4d6 100644 --- a/tests/test_dataset.py +++ b/tests/test_dataset.py @@ -67,22 +67,41 @@ def _annexed(repo: Path, path: Path) -> bool: # ---- what git-annex stores, and what git carries --------------------------- -def test_save_puts_result_bytes_in_the_annex_and_the_manifest_in_git(repo: Path) -> None: - """The whole storage policy, exercised end to end. The manifest has to - stay a plain git blob: `lc` reads it on clones that have fetched no - annex content at all.""" +def test_save_puts_result_bytes_in_the_annex_and_the_record_in_git(repo: Path) -> None: + """The whole storage policy, exercised end to end. The record has to + stay a plain git blob: ASTRA's checker reads it on clones that have + fetched no annex content at all.""" results = repo / "results" / "baseline" - results.mkdir(parents=True) + record = repo / ".astra" / "records" / "baseline" / "best_fit.json" + for folder in (results, record.parent): + folder.mkdir(parents=True) (results / "best_fit.csv").write_text("a,b\n1,2\n") - (results / ".best_fit.manifest.json").write_text('{"data_version": "abc"}\n') + record.write_text('{"address": "best_fit#baseline"}\n') - assert dataset.save(repo, [results], "materialize best_fit") + assert dataset.save(repo, [results, record], "materialize best_fit") assert _annexed(repo, results / "best_fit.csv") - assert not _annexed(repo, results / ".best_fit.manifest.json") + assert not _annexed(repo, record) assert not dataset.status(repo) +def test_a_nested_analysis_has_the_same_policy(repo: Path) -> None: + """An analysis in a folder below the project owns a `results/` and + records of its own: its bytes must not land in git for being one + level down, nor its records in the annex.""" + record = repo / "stage" / ".astra" / "records" / "v1" / "fit.json" + nested = repo / "stage" / "results" / "v1" + for folder in (record.parent, nested): + folder.mkdir(parents=True) + record.write_text('{"address": "fit#v1"}\n') + (nested / "fit.csv").write_text("a,b\n1,2\n") + + assert dataset.save(repo, [repo / "stage"], "a nested result") + + assert not _annexed(repo, record) + assert _annexed(repo, nested / "fit.csv") + + def test_a_plain_git_add_annexes_content_by_itself(repo: Path) -> None: """`filter=annex` is what makes git's own add route content, which is what lets lc — and everyone else — never run a git-annex command.""" @@ -103,7 +122,7 @@ def test_dot_paths_follow_the_storage_policy_not_annex_defaults(repo: Path) -> N writes into results/, would land as a full blob in git, silently, and every clone would carry the bytes forever. `save` opts in unconditionally, so the attributes alone decide: archives and dot - outputs reach the annex, dot-named manifests keep their exemption. + outputs reach the annex. The mutation check is a *plain* `git add` of the same shape, which keeps git-annex's stock dotfile behavior.""" archive = repo / ".datalad" / "environments" / "lc-env-abc" / "image" @@ -112,13 +131,11 @@ def test_dot_paths_follow_the_storage_policy_not_annex_defaults(repo: Path) -> N output = repo / "results" / "baseline" output.mkdir(parents=True) (output / ".cache.h5").write_bytes(b"intermediate\n" * 64) - (output / ".fit.manifest.json").write_text("{}\n") dataset.save(repo, [archive.parent, output], "routed") assert _annexed(repo, archive) assert _annexed(repo, output / ".cache.h5") - assert not _annexed(repo, output / ".fit.manifest.json") plain = repo / ".datalad" / "environments" / "lc-env-def" / "image" plain.parent.mkdir(parents=True) @@ -207,10 +224,7 @@ def test_content_that_is_not_here_is_refused_thin_or_not(repo: Path) -> None: for name in ("thin.bin", "fat.bin"): dataset._git(["annex", "drop", "--force", "--", name], cwd=output) with pytest.raises(assets.ContentNotFetchedError, match="git annex get"): - assets.data_version(output / name) - - with pytest.raises(assets.ContentNotFetchedError): - assets.data_version(output) + assets.require_fetched(output / name) def test_a_locked_file_without_its_content_is_refused_too(repo: Path) -> None: @@ -227,7 +241,7 @@ def test_a_locked_file_without_its_content_is_refused_too(repo: Path) -> None: assert (output / "fit.csv").is_symlink() assert not (output / "fit.csv").exists() with pytest.raises(assets.ContentNotFetchedError, match="git annex get"): - assets.data_version(output) + assets.require_fetched(output / "fit.csv") # ---- committing ------------------------------------------------------------ @@ -289,21 +303,6 @@ def test_save_stages_what_a_rebuild_deleted(repo: Path) -> None: # ---- leaving the tree as clean as it was found ----------------------------- -def test_tracked_manifests_are_only_the_ones_under_results(repo: Path) -> None: - """`results/` is the only place lc writes one; elsewhere the name is - the user's to use, and handing such a file to `assets.read` would ask - it to parse something that was never a manifest.""" - ours = repo / "results" / "baseline" / ".fit.manifest.json" - ours.parent.mkdir(parents=True) - ours.write_text("{}\n") - theirs = repo / "data" / ".vendor.manifest.json" - theirs.parent.mkdir(parents=True, exist_ok=True) - theirs.write_text("[1, 2, 3]\n") - dataset.save(repo, [repo / "results", repo / "data"], "both") - - assert dataset.tracked_manifests(repo) == [ours] - - def test_restore_undoes_a_half_written_rebuild(repo: Path) -> None: """The invariant that makes the dirty-tree refusal survivable: a recipe that truncates a committed output and then fails must not leave the @@ -534,12 +533,14 @@ def test_the_worker_and_the_shim_are_never_console_scripts() -> None: commits nothing, and leaves the tree dirty by design; the shim is the sandbox's own plumbing. A `[project.scripts]` entry would put either on `$PATH` through `uv tool install` — a footgun `lc --help` already - refuses to advertise. Every entry point is either the CLI or a + refuses to advertise. Every console script is either the CLI or a mirrored git-annex executable, and nothing else.""" from importlib.metadata import distribution theirs = {e.value for e in distribution("git-annex").entry_points} - for entry in distribution("lightcone-cli").entry_points: + scripts = distribution("lightcone-cli").entry_points.select(group="console_scripts") + assert scripts + for entry in scripts: assert "lightcone.engine" not in entry.value assert "_sandbox_exec" not in entry.value assert entry.value.startswith("lightcone.cli") or entry.value in theirs, entry diff --git a/tests/test_gpu_execution.py b/tests/test_gpu_execution.py index 4ecf70b3..ddffc871 100644 --- a/tests/test_gpu_execution.py +++ b/tests/test_gpu_execution.py @@ -12,17 +12,15 @@ import pytest -from lightcone.engine import assets, container, identity, plan, worker +from lightcone.engine import container, dataset, identity, plan, worker from lightcone.engine import run as engine_run from lightcone.engine.project import ProjectError from lightcone.engine.sandbox import policy as policy_module _SPEC = """ -version: "0.0.13" -name: analysis -inputs: [] +spec_version: "0.1.0" outputs: - - id: visibility + visibility: type: metric format: txt recipe: @@ -33,13 +31,11 @@ @pytest.fixture def project(analysis: Callable[..., Path]) -> tuple[Path, plan.Task, worker.RunContext]: root = analysis(_SPEC) - task = plan.build(root).tasks[("baseline", "visibility")] + task = plan.build(root).tasks["visibility"] context = worker.RunContext( - env_version=identity.env_version(root), - head=("0123456789abcdef", "https://example/analysis.git"), - versions=assets.Versions(), + environment=identity.environment(root), + commit=dataset.head(root)[0], runtime=container.runtime_for_run(root, build=False), - uv_version="0.0.0-test", ) return root, task, context @@ -54,45 +50,47 @@ def test_recipe_inherits_the_whole_mask_without_changing_the_worker_environment( monkeypatch.setattr(policy_module, "_gpu_device_paths", lambda: ()) result = worker.execute(root, replace(task, resources={"gpus": requested}), {}, context) assert result.status == "ok", result.reason - assert task.output_path.read_text() == ("2,0" if requested else "") + assert (root / task.places[0].bytes).read_text() == ("2,0" if requested else "") assert os.environ["CUDA_VISIBLE_DEVICES"] == "2,0" @pytest.mark.parametrize("mask", [None, ""]) -def test_missing_worker_gpu_mask_preserves_existing_output_and_manifest( +def test_missing_worker_gpu_mask_preserves_existing_output_and_record( project: tuple[Path, plan.Task, worker.RunContext], monkeypatch: pytest.MonkeyPatch, mask: str | None, ) -> None: root, task, context = project - task.output_path.parent.mkdir(parents=True, exist_ok=True) - task.output_path.write_text("previous output") - task.manifest_path.write_text("previous manifest") + made, record = root / task.places[0].bytes, root / task.places[0].record + made.write_text("previous output") + record.parent.mkdir(parents=True, exist_ok=True) + record.write_text("previous record") monkeypatch.delenv("CUDA_VISIBLE_DEVICES", raising=False) if mask is not None: monkeypatch.setenv("CUDA_VISIBLE_DEVICES", mask) with pytest.raises(ProjectError, match="nonempty CUDA_VISIBLE_DEVICES"): worker.execute(root, replace(task, resources={"gpus": 1}), {}, context) - assert task.output_path.read_text() == "previous output" - assert task.manifest_path.read_text() == "previous manifest" + assert made.read_text() == "previous output" + assert record.read_text() == "previous record" @pytest.mark.parametrize("runtime", ["docker", "podman"]) -def test_unsupported_gpu_container_preserves_existing_output_and_manifest( +def test_unsupported_gpu_container_preserves_existing_output_and_record( project: tuple[Path, plan.Task, worker.RunContext], monkeypatch: pytest.MonkeyPatch, runtime: str, ) -> None: root, task, context = project - task.output_path.parent.mkdir(parents=True, exist_ok=True) - task.output_path.write_text("previous output") - task.manifest_path.write_text("previous manifest") + made, record = root / task.places[0].bytes, root / task.places[0].record + made.write_text("previous output") + record.parent.mkdir(parents=True, exist_ok=True) + record.write_text("previous record") monkeypatch.setenv("CUDA_VISIBLE_DEVICES", "2,0") context = replace(context, runtime=replace( context.runtime, mode="containerized", runtime=runtime, )) with pytest.raises(ProjectError, match="GPU containers require podman-hpc"): worker.execute(root, replace(task, resources={"gpus": 1}), {}, context) - assert task.output_path.read_text() == "previous output" - assert task.manifest_path.read_text() == "previous manifest" + assert made.read_text() == "previous output" + assert record.read_text() == "previous record" @pytest.mark.parametrize("use_gpus", [False, True]) @@ -189,7 +187,7 @@ def test_gpu_rerun_missing_mask_explains_how_to_select_local_devices( )) monkeypatch.chdir(root) monkeypatch.delenv("CUDA_VISIBLE_DEVICES", raising=False) - assert worker.main(["baseline/visibility"]) == 2 + assert worker.main(["visibility"]) == 2 error = capsys.readouterr().err assert "CUDA_VISIBLE_DEVICES=0 datalad rerun" in error assert "Slurm sets it" in error diff --git a/tests/test_identity.py b/tests/test_identity.py index 707a23cf..509c26bb 100644 --- a/tests/test_identity.py +++ b/tests/test_identity.py @@ -13,7 +13,7 @@ import pytest -from lightcone.engine.identity import LockScan, definition_version, env_version, scan_lock +from lightcone.engine.identity import LockScan, env_version, environment, scan_lock from lightcone.engine.project import ProjectError _LOCK = """version = 1 @@ -168,48 +168,41 @@ def test_fields_cannot_shift_into_one_another(root: Path) -> None: assert env_version(root) != shifted_left -# ---- definition_version ---------------------------------------------------- +# ---- the environment a record names ----------------------------------------- -def test_definition_version_follows_all_three_terms() -> None: - recipe, decisions = "python fit.py {output}", {"method": "mcmc"} - original = definition_version(recipe=recipe, decisions=decisions, fmt="csv") +def test_a_record_names_the_environment_by_its_hash(root: Path) -> None: + """The IRI is `env_version` under lc's own prefix, so it moves exactly + when the environment does — which is what makes a result *behind*.""" + assert environment(root) == f"urn:lightcone:env:{env_version(root)}" + before = environment(root) + (root / ".python-version").write_text("3.12.9\n") + assert environment(root) != before - assert definition_version(recipe=recipe + " -v", decisions=decisions, fmt="csv") != original - assert ( - definition_version(recipe=recipe, decisions={"method": "nested"}, fmt="csv") != original - ) - assert definition_version(recipe=recipe, decisions=decisions, fmt="parquet") != original - - -def test_the_format_moves_it_even_when_the_recipe_never_names_the_output() -> None: - """A recipe need not use `{output}` at all, so the format cannot be - left to arrive through the rendered path. One that did would keep its - digest across a re-declared serialization — and the sidecar, named from - the id alone, does not move either, so the manifest would go on - describing an output at a path that no longer exists.""" - fixed = "python fit.py --out results/baseline/fit.csv" - assert definition_version(recipe=fixed, decisions={}, fmt="csv") != definition_version( - recipe=fixed, decisions={}, fmt="parquet" - ) +def test_the_iri_is_one_a_record_may_hold(root: Path) -> None: + """ASTRA's record schema is closed and its `iri` has a pattern: an + environment it refuses would leave every result lc makes `missing`.""" + import json + import re + from importlib.resources import files -def test_decision_order_does_not_matter() -> None: - """Decisions are a mapping, not a sequence — two spellings of the same - choices define the same output.""" - a = definition_version(recipe="r", decisions={"x": "1", "y": "2"}, fmt="csv") - b = definition_version(recipe="r", decisions={"y": "2", "x": "1"}, fmt="csv") - assert a == b + schema = json.loads( + (files("astra.schema") / "generated" / "provenance.schema.json").read_text() + ) + pattern = schema["$defs"]["Environment"]["properties"]["iri"]["pattern"] + assert re.fullmatch(pattern, environment(root)) -def test_the_environment_is_not_part_of_what_an_output_is(root: Path) -> None: - """The load-bearing separation. `env_version` is recorded beside an - output and compared to say it is *behind*; folding it in here would - make one added dependency remake a project's every result.""" - before = definition_version(recipe="r", decisions={}, fmt="csv") - (root / ".python-version").write_text("3.12.9\n") - assert env_version(root) # the environment did move - assert definition_version(recipe="r", decisions={}, fmt="csv") == before +def test_astra_check_is_told_the_environment_can_be_recreated(root: Path) -> None: + """The scheme is found through the entry point, the way `astra check` + finds it, and answers only for IRIs lc records.""" + from astra import environments + + answers = environments.answers(environment(root), environments.registered()) + assert [(answer.scheme, answer.recreatable) for answer in answers] == [("lightcone", True)] + assert "uv.lock" in answers[0].holds + assert environments.answers("urn:example:env:1", environments.registered()) == () # ---- the lock scan --------------------------------------------------------- diff --git a/tests/test_materialize.py b/tests/test_materialize.py index 44b323af..be253d1b 100644 --- a/tests/test_materialize.py +++ b/tests/test_materialize.py @@ -13,7 +13,6 @@ from __future__ import annotations import json -import pickle import subprocess import sys from collections.abc import Callable, Iterator @@ -30,44 +29,51 @@ from lightcone.engine.worker import TaskResult _SPEC = """ -version: "0.0.13" -name: analysis +spec_version: "0.1.0" -inputs: - - id: catalog - type: data - source: data/catalog.fits +decisions: + method: + options: {alpha: {}, beta: {}} outputs: - - id: first + first: type: metric format: txt decisions: [method] recipe: command: echo {decisions.method} > {output} - - id: second + second: type: report format: txt inputs: [first] recipe: command: cat {inputs.first} > {output} -decisions: - method: - label: Method - default: alpha - options: - alpha: {label: alpha} - beta: {label: beta} +universes: + baseline: + decisions: {method: alpha} """ -_UNIVERSE = "id: baseline\ndecisions:\n method: alpha\n" - @pytest.fixture def root(analysis: Callable[..., Path]) -> Path: - return analysis(_SPEC, universes={"baseline": _UNIVERSE}) + return analysis(_SPEC) + + +def _switch_method(root: Path) -> None: + """Have the one universe select the other option, and commit it.""" + spec = root / "astra.yaml" + spec.write_text(spec.read_text().replace("{method: alpha}", "{method: beta}")) + dataset.save(root, [spec], "switch method") + + +def _record(root: Path, name: str) -> dict[str, Any]: + """ASTRA's record of the result called *name* in the one universe.""" + record: dict[str, Any] = json.loads( + (root / ".astra/records/baseline" / f"{name}.json").read_text() + ) + return record def _commits(root: Path) -> int: @@ -92,37 +98,39 @@ def test_every_output_is_made_and_committed(root: Path, inline: None) -> None: report = engine.materialize(root, [], cluster_id=CLUSTER_ID) - assert report.made == ["baseline/first", "baseline/second"] + assert report.made == ["first#baseline", "second#baseline"] assert report.ok and not report.up_to_date assert (root / "results/baseline/second.txt").read_text() == "alpha\n" assert _commits(root) == before + 2 assert not dataset.status(root) -def test_an_output_and_its_manifest_land_in_one_commit(root: Path, inline: None) -> None: +def test_an_output_and_its_record_land_in_one_commit(root: Path, inline: None) -> None: """One commit is one complete, self-describing materialization — the - manifest can never come to describe different bytes.""" + record can never come to describe different bytes.""" engine.materialize(root, ["first"], cluster_id=CLUSTER_ID) committed = dataset._git( ["show", "--name-only", "--format=", "HEAD"], cwd=root ).split() assert sorted(committed) == [ - "results/baseline/.first.manifest.json", + ".astra/records/baseline/first.json", "results/baseline/first.txt", ] -def test_the_bytes_go_to_the_annex_and_the_manifest_to_git(root: Path, inline: None) -> None: +def test_the_bytes_go_to_the_annex_and_what_describes_them_to_git( + root: Path, inline: None +) -> None: """What git records is the test: a pointer for content, the real thing - for a manifest. The working tree looks the same either way now.""" + for a record. The working tree looks the same either way.""" engine.materialize(root, ["first"], cluster_id=CLUSTER_ID) def blob(rel: str) -> str: return dataset._git(["cat-file", "-p", f"HEAD:{rel}"], cwd=root) assert blob("results/baseline/first.txt").startswith("/annex/objects/") - assert blob("results/baseline/.first.manifest.json").startswith("{") + assert blob(".astra/records/baseline/first.json").startswith("{") def test_a_second_run_does_nothing_and_commits_nothing(root: Path, inline: None) -> None: @@ -132,11 +140,74 @@ def test_a_second_run_does_nothing_and_commits_nothing(root: Path, inline: None) report = engine.materialize(root, [], cluster_id=CLUSTER_ID) assert report.made == [] - assert report.current == ["baseline/first", "baseline/second"] + assert report.current == ["first#baseline", "second#baseline"] assert report.up_to_date assert _commits(root) == after_first +def test_an_output_built_once_has_no_universe_directory( + analysis: Callable[..., Path], inline: None +) -> None: + """Nothing about it varies between universes, so it is one result, + named by its output alone.""" + root = analysis(_FETCH_SPEC, files={"data/catalog.fits": "stars\n"}) + + report = engine.materialize(root, [], cluster_id=CLUSTER_ID) + + assert report.made == ["copy"] + committed = dataset._git(["show", "--name-only", "--format=", "HEAD"], cwd=root).split() + assert sorted(committed) == [ + ".astra/records/copy.json", + "results/copy.txt", + ] + assert [o.output for o in engine.status(root).outputs] == ["copy"] + + +def test_a_driven_analysis_is_built_in_the_projects_universes( + analysis: Callable[..., Path], inline: None +) -> None: + """A universe that sets a declared analysis's decision drives it: the + result belongs to the project, under the address it is read by, while + its recipe runs where it was written to run — in its own folder.""" + stage = """ + spec_version: "0.1.0" + decisions: + dec: + options: {x: {}, y: {}} + outputs: + out: + type: metric + format: txt + decisions: [dec] + recipe: + command: echo {decisions.dec} $(basename "$PWD") > {output} + """ + spec = """ + spec_version: "0.1.0" + analyses: + stage: {path: stage} + outputs: + report: + type: report + format: txt + inputs: [stage.out] + recipe: + command: cat {inputs} > {output} + universes: + u1: + decisions: {stage.dec: x} + """ + root = analysis(spec, files={"stage/astra.yaml": stage}) + + report = engine.materialize(root, [], cluster_id=CLUSTER_ID) + + assert report.made == ["stage.out#u1", "report#u1"] + assert (root / "results/u1/stage.out.txt").read_text() == "x stage\n" + assert (root / "results/u1/report.txt").read_text() == "x stage\n" + assert not dataset.status(root) + assert engine.materialize(root, [], cluster_id=CLUSTER_ID).up_to_date + + # ---- behind: the environment moved, the analysis did not ------------------- @@ -168,8 +239,8 @@ def test_a_moved_environment_is_reported_and_nothing_is_remade( report = engine.materialize(root, [], cluster_id=CLUSTER_ID) assert report.made == [] - assert set(report.behind) == {"baseline/first", "baseline/second"} - assert "earlier environment" in report.behind["baseline/first"] + assert set(report.behind) == {"first#baseline", "second#baseline"} + assert "earlier environment" in report.behind["first#baseline"] assert report.up_to_date, "behind is not out of date" assert _commits(root) == after_first + 1, "only the environment edit" assert (root / "results/baseline/first.txt").read_text() == first @@ -184,27 +255,11 @@ def test_refresh_remakes_what_is_behind_and_commits_it(root: Path, inline: None) report = engine.materialize(root, [], refresh=True, cluster_id=CLUSTER_ID) - assert set(report.made) == {"baseline/first", "baseline/second"} + assert set(report.made) == {"first#baseline", "second#baseline"} assert report.behind == {} assert _commits(root) == before + 2 - manifest = assets.read(root / "results/baseline/.first.manifest.json") - assert manifest is not None - assert manifest.env_version == identity.env_version(root) - - -def test_the_manifest_records_the_uv_that_converged_the_environment( - root: Path, inline: None -) -> None: - """Probed once by the driver and handed to every task — attestation - beside lc_version, never a rebuild signal.""" - from lightcone.engine import project - - engine.materialize(root, ["first"], cluster_id=CLUSTER_ID) - - manifest = assets.read(root / "results/baseline/.first.manifest.json") - assert manifest is not None - assert manifest.uv_version == project.uv_version(root) - assert manifest.uv_version.count(".") >= 1, "a real version token, not prose" + made_under = _record(root, "first")["generated_by"]["environment"]["iri"] + assert made_under == identity.environment(root) def test_check_reports_behind_without_planning_it(root: Path, inline: None) -> None: @@ -216,7 +271,7 @@ def test_check_reports_behind_without_planning_it(root: Path, inline: None) -> N report = engine.check(root, []) assert report.planned == {} - assert set(report.behind) == {"baseline/first", "baseline/second"} + assert set(report.behind) == {"first#baseline", "second#baseline"} assert report.up_to_date @@ -226,7 +281,7 @@ def test_check_with_refresh_plans_what_is_behind(root: Path, inline: None) -> No report = engine.check(root, [], refresh=True) - assert set(report.planned) == {"baseline/first", "baseline/second"} + assert set(report.planned) == {"first#baseline", "second#baseline"} assert report.behind == {} assert not report.up_to_date @@ -238,15 +293,12 @@ def test_a_stale_output_is_stale_even_when_the_environment_also_moved( here would say "left alone" about something the next run will remake.""" engine.materialize(root, [], cluster_id=CLUSTER_ID) _move_the_environment(root) - (root / "universes" / "baseline.yaml").write_text( - "id: baseline\ndecisions:\n method: beta\n" - ) - dataset.save(root, [root], "switch method") + _switch_method(root) report = engine.check(root, []) - assert "the recipe or its decisions" in report.planned["baseline/first"] - assert "baseline/first" not in report.behind + assert "method" in report.planned["first#baseline"] + assert "first#baseline" not in report.behind # ---- lc status ------------------------------------------------------------- @@ -260,7 +312,7 @@ def test_status_names_the_commit_each_output_came_from(root: Path, inline: None) report = engine.status(root) - assert [o.output for o in report.outputs] == ["baseline/first", "baseline/second"] + assert [o.output for o in report.outputs] == ["first#baseline", "second#baseline"] assert all(o.status == "current" for o in report.outputs) # The commit the tree was at when the run *started* — the code that # produced the output, not the commit the run itself went on to make. @@ -289,7 +341,7 @@ def test_status_leaves_a_never_materialized_output_without_a_commit(root: Path) assert report.counts == {"current": 0, "behind": 0, "stale": 2} assert all(o.git_sha == "" and o.data_version == "" for o in report.outputs) - assert "never been materialized" in report.outputs[0].why + assert "no record" in report.outputs[0].why def test_status_does_not_mind_a_dirty_tree(root: Path, inline: None) -> None: @@ -304,22 +356,71 @@ def test_status_does_not_mind_a_dirty_tree(root: Path, inline: None) -> None: def test_asking_for_an_output_makes_what_it_is_made_of(root: Path, inline: None) -> None: report = engine.materialize(root, ["second"], cluster_id=CLUSTER_ID) - assert report.made == ["baseline/first", "baseline/second"] + assert report.made == ["first#baseline", "second#baseline"] def test_a_changed_decision_remakes_the_output_and_its_dependents( root: Path, inline: None ) -> None: engine.materialize(root, [], cluster_id=CLUSTER_ID) - (root / "universes" / "baseline.yaml").write_text("id: baseline\ndecisions:\n method: beta\n") - dataset.save(root, [root], "switch method") + _switch_method(root) report = engine.materialize(root, [], cluster_id=CLUSTER_ID) - assert report.made == ["baseline/first", "baseline/second"] + assert report.made == ["first#baseline", "second#baseline"] assert (root / "results/baseline/second.txt").read_text() == "beta\n" +def _edit_first_recipe(root: Path, command: str) -> None: + """Rewrite how `first` is made, and commit it.""" + spec = root / "astra.yaml" + spec.write_text(spec.read_text().replace("echo {decisions.method}", command)) + dataset.save(root, [spec], "edit the recipe") + + +def test_a_changed_command_remakes_the_output_and_its_dependents( + root: Path, inline: None +) -> None: + engine.materialize(root, [], cluster_id=CLUSTER_ID) + _edit_first_recipe(root, "echo changed") + + assert "command" in engine.check(root, []).planned["first#baseline"] + report = engine.materialize(root, [], cluster_id=CLUSTER_ID) + + assert report.made == ["first#baseline", "second#baseline"] + assert (root / "results/baseline/second.txt").read_text() == "changed\n" + + +def test_a_byte_identical_rebuild_stops_the_cascade(root: Path, inline: None) -> None: + """What check mode cannot know, a run can: the dependent asks again + once its upstream has landed, and the bytes it recorded reading are + the bytes that are there.""" + engine.materialize(root, [], cluster_id=CLUSTER_ID) + _edit_first_recipe(root, "echo {decisions.method}") + assert set(engine.check(root, []).planned) == {"first#baseline", "second#baseline"} + before = _commits(root) + + report = engine.materialize(root, [], cluster_id=CLUSTER_ID) + + assert report.made == ["first#baseline"] + assert report.current == ["second#baseline"] + assert _commits(root) == before + 1 + + +def test_a_changed_input_remakes_what_reads_it( + analysis: Callable[..., Path], inline: None +) -> None: + root = analysis(_FETCH_SPEC, files={"data/catalog.fits": "stars\n"}) + engine.materialize(root, [], cluster_id=CLUSTER_ID) + catalog = root / "data/catalog.fits" + _forge(catalog, "galaxies\n") + dataset.save(root, [catalog], "a new catalog") + + assert "content of" in engine.check(root, []).planned["copy"] + assert engine.materialize(root, [], cluster_id=CLUSTER_ID).made == ["copy"] + assert (root / "results/copy.txt").read_text() == "galaxies\n" + + def test_the_previous_bytes_are_still_there_at_the_previous_commit( root: Path, inline: None ) -> None: @@ -327,8 +428,7 @@ def test_the_previous_bytes_are_still_there_at_the_previous_commit( exact bytes it produced.""" engine.materialize(root, ["first"], cluster_id=CLUSTER_ID) original = dataset._git(["rev-parse", "HEAD"], cwd=root).strip() - (root / "universes" / "baseline.yaml").write_text("id: baseline\ndecisions:\n method: beta\n") - dataset.save(root, [root], "switch method") + _switch_method(root) engine.materialize(root, ["first"], cluster_id=CLUSTER_ID) assert (root / "results/baseline/first.txt").read_text() == "beta\n" @@ -343,8 +443,8 @@ def test_the_previous_bytes_are_still_there_at_the_previous_commit( def test_check_says_what_would_run_and_why(root: Path) -> None: report = engine.check(root, []) - assert set(report.planned) == {"baseline/first", "baseline/second"} - assert "never been materialized" in report.planned["baseline/first"] + assert set(report.planned) == {"first#baseline", "second#baseline"} + assert "no record" in report.planned["first#baseline"] assert not report.up_to_date @@ -360,17 +460,16 @@ def test_check_writes_nothing_and_commits_nothing(root: Path) -> None: def test_check_cascades_through_an_output_it_already_decided_to_rebuild( root: Path, inline: None ) -> None: - """The `None` sentinel. Check mode cannot know whether a rebuild comes - out byte-identical, so it assumes it will not — the one place it is - deliberately more pessimistic than a worker.""" + """Check mode cannot know whether a rebuild comes out byte-identical, + so it assumes it will not — the one place it is deliberately more + pessimistic than a worker.""" engine.materialize(root, [], cluster_id=CLUSTER_ID) - (root / "universes" / "baseline.yaml").write_text("id: baseline\ndecisions:\n method: beta\n") - dataset.save(root, [root], "switch method") + _switch_method(root) report = engine.check(root, []) - assert "the recipe or its decisions" in report.planned["baseline/first"] - assert report.planned["baseline/second"] == "the input `first` changed" + assert "method" in report.planned["first#baseline"] + assert "first#baseline" in report.planned["second#baseline"] def test_check_does_not_refuse_a_dirty_tree(root: Path) -> None: @@ -388,10 +487,11 @@ def test_a_dirty_tree_refuses_and_says_what_to_do_about_each_path( root: Path, inline: None ) -> None: """Two path classes, two opposite remedies: work the researcher owns is - committed, and anything under `results/` is lc's to write.""" + committed, and anything under `results/` or `.astra/` is lc's to write.""" engine.materialize(root, ["first"], cluster_id=CLUSTER_ID) (root / "notes.md").write_text("in progress\n") (root / "results/baseline/stray.txt").write_text("by hand\n") + (root / ".astra/records/baseline/stray.json").write_text("{}\n") with pytest.raises(ProjectError) as raised: engine.materialize(root, [], cluster_id=CLUSTER_ID) @@ -399,6 +499,7 @@ def test_a_dirty_tree_refuses_and_says_what_to_do_about_each_path( message = str(raised.value) assert "commit these" in message and "notes.md" in message assert "discard these" in message and "results/baseline/stray.txt" in message + assert message.index("discard these") < message.index(".astra/records/baseline/stray.json") assert message.index("lc compute down") < message.index("git restore") @@ -423,10 +524,8 @@ def unexpected(*args: object) -> None: def test_a_run_with_nothing_to_make_needs_no_cluster( analysis: Callable[..., Path], monkeypatch: pytest.MonkeyPatch, ) -> None: - root = analysis( - 'version: "0.0.13"\nname: analysis\ninputs: []\noutputs: []\ndecisions: {}\n', - universes={"baseline": "id: baseline\ndecisions: {}\n"}, - ) + # Built per universe, and no universe to build it in. + root = analysis(_SPEC[: _SPEC.index("universes:")]) _declare_license(root) def unexpected(*args: object) -> None: @@ -439,7 +538,7 @@ def unexpected(*args: object) -> None: assert (root / "ro-crate-metadata.json").is_file() -@pytest.mark.parametrize("failing", ["baseline/first", "baseline/second"]) +@pytest.mark.parametrize("failing", ["first#baseline", "second#baseline"]) def test_a_failed_commit_says_whether_remote_tasks_may_still_run( root: Path, monkeypatch: pytest.MonkeyPatch, failing: str, ) -> None: @@ -447,8 +546,7 @@ def test_a_failed_commit_says_whether_remote_tasks_may_still_run( save = dataset.save def fail(root: Path, paths: list[Path], message: str) -> None: - universe, output = failing.split("/") - if message.startswith(engine.datalad_run_subject((universe, output))): + if message.startswith(engine.datalad_run_subject(failing)): raise ProjectError("git commit failed") save(root, paths, message) @@ -456,109 +554,7 @@ def fail(root: Path, paths: list[Path], message: str) -> None: with pytest.raises(ProjectError, match="git commit failed") as raised: engine.materialize(root, [], cluster_id=CLUSTER_ID) # Only the first output leaves another one outstanding. - assert ("did not stop the allocation" in str(raised.value)) == (failing == "baseline/first") - -def test_shared_inputs_are_hashed_once_before_task_serialization( - analysis: Callable[..., Path], monkeypatch: pytest.MonkeyPatch, -) -> None: - spec = _consuming("data/catalog.fits").replace("inputs: [first]", "inputs: [first, catalog]") - root = analysis(spec, files={"data/catalog.fits": "data"}, universes={"baseline": _UNIVERSE}) - hashed: list[Path] = [] - real = assets.data_version - - def digest(path: Path) -> str: - hashed.append(path) - return real(path) - - class _Copied(_Inline): - def submit( - self, fn: Callable[..., object], *args: object, key: str, resources: dict[str, float], - ) -> object: - return fn(*pickle.loads(pickle.dumps(args))) - - monkeypatch.setattr(assets, "data_version", digest) - _cluster(monkeypatch, _Copied()) - first = engine.materialize(root, [], cluster_id=CLUSTER_ID) - assert len(first.made) == 2 - assert hashed.count(root / "data/catalog.fits") == 1 - second = engine.materialize(root, [], cluster_id=CLUSTER_ID) - assert len(second.current) == 2 - assert hashed.count(root / "data/catalog.fits") == 2 - - -@pytest.mark.parametrize("kind", ["missing", "symlink-loop"]) -def test_an_unreadable_input_fails_only_its_dependent_tasks( - analysis: Callable[..., Path], inline: None, tmp_path: Path, kind: str, -) -> None: - source = "data/missing.fits" - if kind == "symlink-loop": - loop = tmp_path / "loop" - loop.symlink_to(loop.name) - source = str(loop) - spec = _consuming(source).replace("decisions:\n method:", """ - - id: independent - type: metric - format: txt - recipe: - command: echo done > {output} - -decisions: - method:""") - root = analysis(spec, universes={"baseline": _UNIVERSE}) - report = engine.materialize(root, [], cluster_id=CLUSTER_ID) - assert report.failed == ["baseline/first"] - assert report.blocked == ["baseline/second"] - assert report.made == ["baseline/independent"] - assert not dataset.status(root) - - -def _consuming(source: str) -> str: - """`_SPEC` with `first` actually reading the declared input, from *source*.""" - return _SPEC.replace("source: data/catalog.fits", f"source: {source}").replace( - " decisions: [method]", " inputs: [catalog]\n decisions: [method]" - ) - - -def test_an_unreadable_declared_input_does_not_traceback_out_of_a_read_only_verb( - analysis: Callable[..., Path], -) -> None: - """A declared input directory can hold a symlink pointing nowhere — - the directory walk keeps dangling links deliberately, so that an - unfetched annexed file cannot silently drop out of the digest. One that - is not an annex link then reaches `open()`. `status` and `--check` read - projects that are in a state, so neither may raise.""" - root = analysis(_consuming("data/inputs"), universes={"baseline": _UNIVERSE}) - (root / "data" / "inputs").mkdir() - (root / "data" / "inputs" / "broken").symlink_to("nowhere.fits") - - assert engine.status(root).outputs - assert engine.check(root, []).planned - - -def test_a_declared_input_outside_the_project_is_reported_as_unrecoverable( - analysis: Callable[..., Path], tmp_path: Path -) -> None: - """Its bytes are hashed into the manifest like any other input, so a - change to it still cascades — but it is not in the repository, so the - commit that records the output cannot bring it back.""" - outside = tmp_path / "shared" / "catalog.fits" - outside.parent.mkdir() - outside.write_text("elsewhere\n") - root = analysis(_consuming(str(outside)), universes={"baseline": _UNIVERSE}) - - report = engine.check(root, []) - - assert any(str(outside) in w and "cannot restore them" in w for w in report.warnings) - - -def test_a_declared_input_inside_the_project_draws_no_such_warning( - analysis: Callable[..., Path], -) -> None: - """The mutation check on the test above: the same spec with the source - back under `data/` says nothing.""" - root = analysis(_consuming("data/catalog.fits"), universes={"baseline": _UNIVERSE}) - - assert not any("cannot restore" in w for w in engine.check(root, []).warnings) + assert ("did not stop the allocation" in str(raised.value)) == (failing == "first#baseline") def test_a_dependency_the_lock_does_not_pin_is_refused(root: Path) -> None: @@ -621,11 +617,10 @@ def test_ambient_uv_settings_are_scrubbed_and_reported( def test_an_edit_while_the_graph_runs_is_reported( root: Path, monkeypatch: pytest.MonkeyPatch ) -> None: - """The dirty check runs at start of run and manifests are written - per-output later, so an edit in between leaves manifests whose - git_sha no longer describes the code that ran. The run ends with one - status call and says so — the honest floor under the unwritten - `git_dirty` field.""" + """The dirty check runs at start of run and records are written + per-output later, so an edit in between leaves records whose commit + no longer describes the code that ran. The run ends with one status + call and says so.""" class Editing(_Inline): def completed(self, handles: list[object]) -> Iterator[object]: @@ -680,16 +675,15 @@ def test_a_failing_recipe_commits_nothing_and_leaves_the_tree_clean( analysis: Callable[..., Path], inline: None ) -> None: """The invariant that makes the dirty-tree refusal survivable: the next - run must not tell the user to commit truncated, manifest-less + run must not tell the user to commit truncated, record-less garbage.""" - spec = _SPEC.replace("echo {decisions.method} > {output}", "exit 1") - root = analysis(spec, universes={"baseline": _UNIVERSE}) + root = analysis(_SPEC.replace("echo {decisions.method} > {output}", "exit 1")) before = _commits(root) report = engine.materialize(root, [], cluster_id=CLUSTER_ID) - assert report.failed == ["baseline/first"] - assert report.blocked == ["baseline/second"] + assert report.failed == ["first#baseline"] + assert report.blocked == ["second#baseline"] assert not report.ok assert _commits(root) == before assert not dataset.status(root) @@ -701,8 +695,7 @@ def test_a_run_in_which_everything_failed_is_not_up_to_date( """`made` stays empty when every recipe fails, so `up_to_date` alone read "nothing to do" over a list of failures — and it is the second key of the JSON report, which is what an agent branches on.""" - spec = _SPEC.replace("echo {decisions.method} > {output}", "exit 1") - root = analysis(spec, universes={"baseline": _UNIVERSE}) + root = analysis(_SPEC.replace("echo {decisions.method} > {output}", "exit 1")) report = engine.materialize(root, [], cluster_id=CLUSTER_ID) @@ -725,7 +718,7 @@ def test_a_rebuild_that_fails_puts_the_previous_output_back( report = engine.materialize(root, ["first"], cluster_id=CLUSTER_ID) - assert report.failed == ["baseline/first"] + assert report.failed == ["first#baseline"] assert (root / "results/baseline/first.txt").read_text() == "alpha\n" assert _commits(root) == at_break assert not dataset.status(root) @@ -773,18 +766,18 @@ def test_the_run_record_is_what_datalad_reads( subject, info = get_run_info(Dataset(str(root)), message) - assert subject == "second [baseline]" + assert subject == "second#baseline" assert info is not None # Full-string, not endswith: a flag appearing or vanishing here must # fail this test, not slip past a suffix match. assert info["cmd"] == ( "uv run --no-project --with 'lightcone-cli==1.2.3' -- " - "python -m lightcone.engine.worker baseline/second" + "python -m lightcone.engine.worker 'second#baseline'" ) assert info["inputs"] == ["results/baseline/first.txt"] assert info["outputs"] == [ "results/baseline/second.txt", - "results/baseline/.second.manifest.json", + ".astra/records/baseline/second.json", ] assert info["dsid"] == "4b7b5c1e-0000-4000-8000-000000000000" assert info["chain"] == [] and info["pwd"] == "." @@ -813,26 +806,10 @@ def test_the_engine_pin_follows_the_build(monkeypatch: pytest.MonkeyPatch) -> No def test_the_record_names_the_declared_input_not_the_annex_object( analysis: Callable[..., Path], inline: None ) -> None: - """Declared inputs are annex symlinks, so a resolved path records + """A declared input is annexed, so its bytes live under `.git/annex/objects/SHA256E-…` — the storage rather than the input, and something no one can `datalad get`.""" - spec = """ - version: "0.0.13" - name: analysis - inputs: - - id: catalog - type: data - source: data/catalog.txt - outputs: - - id: fit - type: metric - format: txt - inputs: [catalog] - recipe: - command: cat {inputs.catalog} > {output} - """ - root = analysis(spec, files={"data/catalog.txt": "measured\n"}) - dataset.save(root, [root / "data"], "the catalog") + root = analysis(_FETCH_SPEC, files={"data/catalog.fits": "measured\n"}) engine.materialize(root, [], cluster_id=CLUSTER_ID) @@ -841,28 +818,43 @@ def test_the_record_names_the_declared_input_not_the_annex_object( _, info = get_run_info(Dataset(str(root)), dataset._git(["log", "-1", "--format=%B"], cwd=root)) assert info is not None - assert info["inputs"] == ["data/catalog.txt"] + assert info["inputs"] == ["data/catalog.fits"] -def test_every_manifest_of_one_run_names_the_same_commit(root: Path, inline: None) -> None: +def test_every_record_of_one_run_names_the_commit_it_started_from( + root: Path, inline: None +) -> None: """The driver commits each output as it lands, so HEAD moves during the - run — and reading it per task would stamp later manifests with a commit + run — and reading it per task would stamp later records with a commit this same run created, nondeterministically.""" + started_at = dataset.head(root)[0] + + engine.materialize(root, [], cluster_id=CLUSTER_ID) + + for name in ("first", "second"): + assert _record(root, name)["generated_by"]["commit"] == started_at + + +def test_a_record_says_what_was_made_from_what_and_by_whom(root: Path, inline: None) -> None: + """The record is ASTRA's, and it is what every later check compares the + project against — so what the run read is the upstream's own bytes, and + the environment is the one `behind` is measured from.""" engine.materialize(root, [], cluster_id=CLUSTER_ID) - shas = { - assets.read( # type: ignore[union-attr] - root / "results/baseline" / f".{name}.manifest.json" - ).git_sha - for name in ("first", "second") - } - assert len(shas) == 1 + first, second = _record(root, "first"), _record(root, "second") + + assert second["address"] == "second#baseline" + assert second["generated_by"]["inputs"] == [ + {"entry": "first", "content_hash": first["content_hash"], "size": first["size"]} + ] + assert second["generated_by"]["environment"] == {"iri": identity.environment(root)} + assert second["generated_by"]["runner"]["id"] == "lightcone-cli" def test_check_agrees_with_a_run_on_a_clone_with_no_annex_content( root: Path, inline: None, tmp_path: Path ) -> None: - """Manifests are in git, so an output whose bytes were never fetched + """Records are in git, so an output whose bytes were never fetched is still classifiable — check mode reads the recorded digest rather than the pointer file sitting in its place.""" engine.materialize(root, [], cluster_id=CLUSTER_ID) @@ -891,16 +883,14 @@ def _clone(root: Path, into: Path) -> Path: _FETCH_SPEC = """ -version: "0.0.13" -name: analysis +spec_version: "0.1.0" inputs: - - id: catalog - type: data - source: data/catalog.fits + catalog: + path: data/catalog.fits outputs: - - id: copy + copy: type: metric format: txt inputs: [catalog] @@ -913,20 +903,19 @@ def test_a_bytes_free_clone_fetches_its_inputs_and_is_up_to_date( analysis: Callable[..., Path], inline: None, tmp_path: Path ) -> None: """lc fetches declared inputs rather than telling anyone to — a clone - holding only pointers materializes straight to up-to-date, hashing - the same bytes the origin recorded. Without the fetch this run - *failed*: the worker's hash refused the pointer file.""" + holding only pointers materializes straight to up-to-date, and ends + holding the bytes a recipe would read.""" root = analysis(_FETCH_SPEC, files={"data/catalog.fits": "stars\n"}) assert engine.materialize(root, [], cluster_id=CLUSTER_ID).ok clone = _clone(root, tmp_path) with pytest.raises(assets.ContentNotFetchedError): - assets.data_version(clone / "data" / "catalog.fits") + assets.require_fetched(clone / "data" / "catalog.fits") again = engine.materialize(clone, [], cluster_id=CLUSTER_ID) assert again.up_to_date, (again.failed, again.warnings) - assert assets.data_version(clone / "data" / "catalog.fits") # the bytes came + assert (clone / "data" / "catalog.fits").read_text() == "stars\n" # the bytes came assert not dataset.status(clone) @@ -935,17 +924,20 @@ def test_an_unreachable_input_is_a_warning_and_a_per_task_failure( ) -> None: """A failed fetch must not refuse the whole run — independent tasks still run, and the task whose input is unreachable reports its own - failure. Reaching the state honestly: clone, then delete the origin - the annex would fetch from.""" + failure. Reaching the state honestly: clone, delete the origin the + annex would fetch from, and give the recipe a reason to run again.""" root = analysis(_FETCH_SPEC, files={"data/catalog.fits": "stars\n"}) assert engine.materialize(root, [], cluster_id=CLUSTER_ID).ok clone = _clone(root, tmp_path) dataset._git(["remote", "remove", "origin"], cwd=clone) + spec = clone / "astra.yaml" + spec.write_text(spec.read_text().replace("cat {inputs.catalog}", "cat -u {inputs.catalog}")) + dataset.save(clone, [spec], "edit the recipe") report = engine.materialize(clone, [], cluster_id=CLUSTER_ID) assert not report.ok - assert report.failed == ["baseline/copy"] + assert report.failed == ["copy"] assert any("could not be fetched" in w for w in report.warnings) @@ -968,8 +960,8 @@ def test_a_drifted_environment_is_made_to_match_before_anything_runs( where the environment is made to match the lock (a rerun's worker entry point syncs for itself). Reported and refused, a lock edited without a sync would leave recipes importing packages the lock does - not describe while every manifest recorded the new lock's - `env_version`; doing it instead is shorter and impossible to ignore.""" + not describe while every record named the new lock's environment; + doing it instead is shorter and impossible to ignore.""" pyproject = root / "pyproject.toml" pyproject.write_text( pyproject.read_text().replace("dependencies = []", 'dependencies = ["idna"]') @@ -994,7 +986,7 @@ def test_the_recorded_command_reproduces_the_output( ) -> None: """The claim the record makes, run literally. `datalad rerun` removes the output, executes the recorded command, and commits what came - back — so the manifest is regenerated inside the same commit, by the + back — so the record is regenerated inside the same commit, by the pinned engine resolved into an ephemeral environment.""" pytest.importorskip("datalad") version, dist = engine_dist @@ -1003,8 +995,7 @@ def test_the_recorded_command_reproduces_the_output( # working tree — the code actually under test. monkeypatch.setattr(engine, "_engine_requirement", lambda: f"lightcone-cli=={version}") engine.materialize(root, ["first"], cluster_id=CLUSTER_ID) - original = assets.read(root / "results/baseline/.first.manifest.json") - assert original is not None + original = _record(root, "first") proc = subprocess.run( [sys.executable, "-c", "from datalad.api import rerun; rerun('HEAD')"], @@ -1015,10 +1006,8 @@ def test_the_recorded_command_reproduces_the_output( ) assert proc.returncode == 0, proc.stderr - rerun = assets.read(root / "results/baseline/.first.manifest.json") - assert rerun is not None - assert rerun.data_version == original.data_version - assert rerun.data_version == assets.data_version(root / "results/baseline/first.txt") + assert _record(root, "first")["content_hash"] == original["content_hash"] + assert (root / "results/baseline/first.txt").read_text() == "alpha\n" assert not dataset.status(root) @@ -1038,8 +1027,7 @@ def test_the_recorded_command_holds_on_a_fresh_clone( version, dist = engine_dist monkeypatch.setattr(engine, "_engine_requirement", lambda: f"lightcone-cli=={version}") engine.materialize(root, ["first"], cluster_id=CLUSTER_ID) - original = assets.read(root / "results/baseline/.first.manifest.json") - assert original is not None + original = _record(root, "first") clone = _clone(root, tmp_path) assert not (clone / ".venv").exists() @@ -1053,9 +1041,7 @@ def test_the_recorded_command_holds_on_a_fresh_clone( ) assert proc.returncode == 0, proc.stderr - rerun = assets.read(clone / "results/baseline/.first.manifest.json") - assert rerun is not None - assert rerun.data_version == original.data_version + assert _record(clone, "first")["content_hash"] == original["content_hash"] assert (clone / ".venv").exists() @@ -1087,7 +1073,7 @@ def test_resource_refusal_precedes_preparation_and_all_recipes( spec = _SPEC.replace( "command: cat", f"resources: {{{resource_spec}}}\n command: cat" ) - root = analysis(spec, universes={"baseline": _UNIVERSE}) + root = analysis(spec) before = dataset.head(root) _resource_cluster(monkeypatch) @@ -1096,7 +1082,7 @@ def unexpected(*args: object, **kwargs: object) -> None: monkeypatch.setattr(engine, "_fetch_inputs", unexpected) monkeypatch.setattr(engine.container, "runtime_for_run", unexpected) - with pytest.raises(ProjectError, match="baseline/second:.*no worker"): + with pytest.raises(ProjectError, match="second#baseline:.*no worker"): engine.materialize(root, [], cluster_id=CLUSTER_ID) assert dataset.head(root) == before assert not dataset.status(root) @@ -1104,7 +1090,7 @@ def unexpected(*args: object, **kwargs: object) -> None: @pytest.mark.parametrize("resource_spec", [ - "gpus: 1", "disk: 1Gi", "cpus: 0.5", "cpus: 0", "memory: null", "time_limit: 1s", + "gpus: 1", "disk: 1Gi", "cpus: 0.5", "cpus: 0", "time_limit: 1s", ]) def test_execution_only_resources_do_not_block_read_only_commands( analysis: Callable[..., Path], monkeypatch: pytest.MonkeyPatch, resource_spec: str, @@ -1112,9 +1098,9 @@ def test_execution_only_resources_do_not_block_read_only_commands( spec = _SPEC.replace( "command: cat", f"resources: {{{resource_spec}}}\n command: cat", ) - root = analysis(spec, universes={"baseline": _UNIVERSE}) + root = analysis(spec) assert len(engine.status(root).outputs) == 2 - assert set(engine.check(root, []).planned) == {"baseline/first", "baseline/second"} + assert set(engine.check(root, []).planned) == {"first#baseline", "second#baseline"} _resource_cluster(monkeypatch) @@ -1122,18 +1108,17 @@ def unexpected(*args: object, **kwargs: object) -> None: pytest.fail("preparation began before execution requirements were validated") monkeypatch.setattr(engine, "_fetch_inputs", unexpected) - with pytest.raises(ProjectError, match="baseline/second"): + with pytest.raises(ProjectError, match="second#baseline"): engine.materialize(root, [], cluster_id=CLUSTER_ID) assert not dataset.status(root) - @pytest.mark.parametrize("runtime_name", ["docker", "podman"]) def test_gpu_runtime_refusal_precedes_image_build_and_all_recipes( analysis: Callable[..., Path], monkeypatch: pytest.MonkeyPatch, runtime_name: str, ) -> None: spec = _SPEC.replace("command: cat", "resources: {gpus: 1}\n command: cat") - root = analysis(spec, universes={"baseline": _UNIVERSE}) + root = analysis(spec) _resource_cluster(monkeypatch, gpus=1) pyproject = root / "pyproject.toml" pyproject.write_text(pyproject.read_text() + "\n[tool.lightcone.image]\napt-install = []\n") @@ -1151,6 +1136,7 @@ def unexpected(*args: object, **kwargs: object) -> None: assert not dataset.status(root) assert not (root / "results/baseline/first.txt").exists() + def test_empty_cluster_refuses_before_project_preparation( root: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: @@ -1180,8 +1166,8 @@ def test_real_dask_respects_recipe_resource_reservations( ) -> None: # Four Dask threads would run all four subprocesses together without # resource reservations. Each independent output records its live interval. - spec = 'version: "0.0.13"\nname: analysis\ninputs: []\noutputs:\n' + "".join( - f" - id: task{index}\n" + spec = 'spec_version: "0.1.0"\noutputs:\n' + "".join( + f" task{index}:\n" " type: metric\n" " format: json\n" " recipe:\n" @@ -1208,7 +1194,7 @@ def test_real_dask_respects_recipe_resource_reservations( assert report.ok and len(report.made) == 4 events = [] - for path in (root / "results/baseline").glob("task*.json"): + for path in (root / "results").glob("task*.json"): start, finish, visible = json.loads(path.read_text()) assert visible == ("2,0" if gpus else "") events.extend([(start, 1), (finish, -1)]) @@ -1220,23 +1206,22 @@ def test_real_dask_respects_recipe_resource_reservations( assert not dataset.status(root) - def test_current_gpu_output_needs_no_gpu_to_build_a_cpu_dependent( analysis: Callable[..., Path], monkeypatch: pytest.MonkeyPatch, ) -> None: spec = _SPEC.replace("command: echo", "resources: {gpus: 1}\n command: echo") - root = analysis(spec, universes={"baseline": _UNIVERSE}) + root = analysis(spec) monkeypatch.setenv("CUDA_VISIBLE_DEVICES", "0") _resource_cluster(monkeypatch, gpus=1) - assert engine.materialize(root, ["first"], cluster_id=CLUSTER_ID).made == ["baseline/first"] - original = (root / "results/baseline/.first.manifest.json").read_bytes() + assert engine.materialize(root, ["first"], cluster_id=CLUSTER_ID).made == ["first#baseline"] + original = (root / ".astra/records/baseline/first.json").read_bytes() _resource_cluster(monkeypatch) report = engine.materialize(root, [], cluster_id=CLUSTER_ID) assert report.ok - assert report.current == ["baseline/first"] - assert report.made == ["baseline/second"] - assert (root / "results/baseline/.first.manifest.json").read_bytes() == original + assert report.current == ["first#baseline"] + assert report.made == ["second#baseline"] + assert (root / ".astra/records/baseline/first.json").read_bytes() == original assert not dataset.status(root) @@ -1256,12 +1241,13 @@ def submit(self, *args: Any, **kwargs: Any) -> Any: _cluster(monkeypatch, NoExecution()) assert len(engine.materialize(root, [], cluster_id=CLUSTER_ID).current) == 2 + def test_a_real_cluster_still_fits_through_the_seam(root: Path, cluster_id: str) -> None: """The one test that starts Dask. The seam is only worth having if the thing it abstracts still goes through it.""" report = engine.materialize(root, [], cluster_id=CLUSTER_ID) - assert report.made == ["baseline/first", "baseline/second"] + assert report.made == ["first#baseline", "second#baseline"] assert not dataset.status(root) @@ -1270,7 +1256,7 @@ def test_a_processes_cluster_fits_through_the_seam( ) -> None: """Workers in other processes — the shape every venue beyond one machine has. Pins that the unit crosses the boundary by reference - (`worker.materialize`, `Task`, `Versions`, `TaskResult`) and that + (`worker.materialize`, `Task`, `RunContext`, `TaskResult`) and that results travel back whole.""" from distributed import Client, LocalCluster @@ -1289,7 +1275,7 @@ def processes(cluster_id: str) -> Iterator[Any]: monkeypatch.setattr(compute, "connect", processes) report = engine.materialize(root, [], cluster_id=CLUSTER_ID) - assert report.made == ["baseline/first", "baseline/second"] + assert report.made == ["first#baseline", "second#baseline"] assert not dataset.status(root) @@ -1301,7 +1287,7 @@ def test_the_report_is_json_ready(root: Path, inline: None) -> None: data = json.loads(json.dumps(report.as_dict())) assert data["ok"] is True - assert data["made"] == ["baseline/first"] + assert data["made"] == ["first#baseline"] # ---- the foreign-write fact ------------------------------------------------ @@ -1335,12 +1321,12 @@ def test_a_foreign_write_is_stale_and_names_its_commit(root: Path, inline: None) outputs = {o.output: o for o in engine.status(root).outputs} - assert outputs["baseline/first"].status == "stale" + assert outputs["first#baseline"].status == "stale" forged_sha = dataset.last_writer(root, root / "results/baseline/first.txt").sha - assert outputs["baseline/first"].foreign_write == forged_sha - assert "tweak colors" in outputs["baseline/first"].why - assert "git show" in outputs["baseline/first"].why - assert not outputs["baseline/second"].foreign_write + assert outputs["first#baseline"].foreign_write == forged_sha + assert "tweak colors" in outputs["first#baseline"].why + assert "git show" in outputs["first#baseline"].why + assert not outputs["second#baseline"].foreign_write def test_check_plans_the_remake_of_a_foreign_written_output( @@ -1355,7 +1341,8 @@ def test_check_plans_the_remake_of_a_foreign_written_output( report = engine.check(root, []) - assert any("tweak colors" in why for why in report.planned.values()) + assert "tweak colors" in report.planned["first#baseline"] + assert "would be remade" in report.planned["second#baseline"] assert not report.up_to_date @@ -1373,10 +1360,10 @@ def test_the_foreign_write_fact_survives_a_bytes_free_clone( outputs = {o.output: o for o in engine.status(clone).outputs} - assert outputs["baseline/first"].status == "stale" - assert "tweak colors" in outputs["baseline/first"].why - assert outputs["baseline/first"].foreign_write - assert not outputs["baseline/second"].foreign_write + assert outputs["first#baseline"].status == "stale" + assert "tweak colors" in outputs["first#baseline"].why + assert outputs["first#baseline"].foreign_write + assert not outputs["second#baseline"].foreign_write def test_the_next_run_remakes_a_foreign_written_output(root: Path, inline: None) -> None: @@ -1390,7 +1377,7 @@ def test_the_next_run_remakes_a_foreign_written_output(root: Path, inline: None) report = engine.materialize(root, [], cluster_id=CLUSTER_ID) - assert "baseline/first" in report.made + assert "first#baseline" in report.made assert forged.read_text() == "alpha\n" # the recipe's bytes, not the hand's assert all(not o.foreign_write for o in engine.status(root).outputs) @@ -1483,19 +1470,17 @@ def test_status_places_the_publication_view(root: Path, inline: None) -> None: def test_status_sees_the_crate_lag_a_rerun_leaves(root: Path, inline: None) -> None: - """The recorded residue made visible: a rerun rewrites a manifest but + """The recorded residue made visible: a rerun rewrites a record but never regenerates the view. Status reads the mismatch off the - document's own datePublished against the manifests it already read — + document's own datePublished against the records it already read — no git, no rocrate import.""" - from dataclasses import replace - _declare_license(root) engine.materialize(root, [], cluster_id=CLUSTER_ID) - sidecar = root / "results/baseline/.second.manifest.json" - manifest = assets.read(sidecar) - assert manifest is not None - assets.write(sidecar, replace(manifest, finished_at="2027-01-01T00:00:00.000+00:00")) - dataset.save(root, [sidecar], "a rerun-shaped manifest rewrite") + path = root / ".astra/records/baseline/second.json" + record = _record(root, "second") + record["generated_by"]["finished"] = "2027-01-01T00:00:00.000Z" + path.write_text(json.dumps(record)) + dataset.save(root, [path], engine.datalad_run_subject("second#baseline")) assert engine.status(root).crate.startswith("behind") @@ -1508,12 +1493,12 @@ def test_an_output_the_spec_dropped_is_excluded_and_named(root: Path, inline: No engine.materialize(root, [], cluster_id=CLUSTER_ID) spec_path = root / "astra.yaml" spec = spec_path.read_text() - block = spec[spec.index(" - id: second") : spec.index("\ndecisions:") + 1] + block = spec[spec.index(" second:") : spec.index("\nuniverses:") + 1] spec_path.write_text(spec.replace(block, "")) dataset.save(root, [spec_path], "drop the second output") report = engine.materialize(root, [], cluster_id=CLUSTER_ID) - assert any(".second.manifest.json" in w for w in report.warnings) document = (root / "ro-crate-metadata.json").read_text() assert "results/baseline/second.txt" not in document + assert any(".astra/records/baseline/second.json" in w for w in report.warnings) diff --git a/tests/test_plan.py b/tests/test_plan.py index 9f069d38..54eb936d 100644 --- a/tests/test_plan.py +++ b/tests/test_plan.py @@ -1,315 +1,368 @@ -"""Tests for `lightcone.engine.plan` — the spec, read as a graph. - -Two things are being pinned here. That the graph says what the spec meant -— which outputs exist, what each depends on, and what its recipe came out -as — and that everything ambiguous is an error rather than a guess: a -placeholder nothing declares, a target nothing matches, a nesting depth -the addressing scheme cannot express. +"""Tests for `lightcone.engine.plan` — ASTRA's plan, carried across as tasks. + +What the spec *means* is ASTRA's and is tested there. What is pinned here +is what lc adds: that each planned result becomes one task holding +everything a worker needs, that every path in it is measured from the +project root — wherever that sits in its repository — and that what this +engine cannot execute, or a target that matches nothing, is an error +rather than a guess. """ from __future__ import annotations +import subprocess import textwrap from pathlib import Path import pytest from lightcone.engine import plan -from lightcone.engine.plan import Graph from lightcone.engine.project import ProjectError _SPEC = """ -version: "0.0.13" -name: demo +spec_version: "0.1.0" + +analyses: + stage: {path: stage} inputs: - - id: catalog - type: data - source: data/catalog.fits + catalog: {path: data/catalog.csv} + +decisions: + method: + options: + mcmc: {} + nested: {} + +universes: + baseline: {decisions: {method: mcmc, stage.depth: deep}} + alternative: {decisions: {method: nested, stage.depth: shallow}} outputs: - - id: fit + summary: + type: table + format: csv + inputs: [catalog] + recipe: + command: python src/summary.py {inputs.catalog} {output} + + fit: type: metric format: json - inputs: [catalog] + inputs: [summary, catalog, stage.mask] decisions: [method] recipe: - command: python src/fit.py {inputs.catalog} --method {decisions.method} {output} + command: python src/fit.py {inputs.summary} {inputs.stage.mask} -m {decisions.method} {output} - - id: report + report: type: report format: md inputs: [fit] recipe: command: python src/report.py {inputs.fit} {output} +""" - - id: reexport - type: metric +#: The same analysis reading one of the stage's own universes instead of +#: driving it, so the stage's result stays the stage's. +_PINNED = ( + _SPEC.replace(", stage.depth: deep", "") + .replace(", stage.depth: shallow", "") + .replace("stage.mask]", "stage.mask#v]") + .replace("{inputs.stage.mask}", "{inputs.stage.mask#v}") +) + +_STAGE = """ +spec_version: "0.1.0" decisions: - method: - label: Method - default: mcmc + depth: options: - mcmc: {label: MCMC} - nested: {label: Nested} -""" + deep: {} + shallow: {} +universes: + v: {decisions: {depth: deep}} -def _project(root: Path, spec: str = _SPEC, **universes: str) -> Path: - (root / "astra.yaml").write_text(textwrap.dedent(spec)) - (root / "universes").mkdir(exist_ok=True) - declared = universes or {"baseline": "id: baseline\ndecisions:\n method: mcmc\n"} - for name, text in declared.items(): - (root / "universes" / f"{name}.yaml").write_text(textwrap.dedent(text)) - return root +outputs: + mask: + type: data + format: txt + decisions: [depth] + recipe: + command: python mask.py {decisions.depth} {output} +""" -def _build(root: Path) -> Graph: - return plan.build(root) +def _git(repo: Path, *argv: str) -> str: + done = subprocess.run( + ["git", "-c", "user.name=Test", "-c", "user.email=t@example.com", *argv], + cwd=repo, + check=True, + capture_output=True, + text=True, + ) + return done.stdout.strip() -# ---- what the graph contains ----------------------------------------------- +def _project(repo: Path, spec: str = _SPEC, *, below: str = "") -> Path: + """Write the analysis and its stage into a repository, tracked. + Args: + repo: The top of the repository to create. + spec: The root analysis. + below: The project root's path from the top of the repository. -def test_one_task_per_universe_and_output_with_a_recipe(tmp_path: Path) -> None: - """A re-export declares no recipe, so it produces no bytes and there is - nothing to schedule for it.""" - graph = _build(_project(tmp_path)) - assert sorted(graph.tasks) == [("baseline", "fit"), ("baseline", "report")] + Returns: + The project root. + """ + root = repo / below + for folder in ("data", "stage"): + (root / folder).mkdir(parents=True) + (root / "astra.yaml").write_text(textwrap.dedent(spec)) + (root / "stage" / "astra.yaml").write_text(textwrap.dedent(_STAGE)) + (root / "data" / "catalog.csv").write_text("x,y\n1,2\n") + _git(repo, "init", "-q") + _git(repo, "add", ".") + return root -def test_every_universe_gets_its_own_task(tmp_path: Path) -> None: - graph = _build( - _project( - tmp_path, - baseline="id: baseline\ndecisions:\n method: mcmc\n", - alternative="id: alternative\ndecisions:\n method: nested\n", - ) +# ---- what a task carries --------------------------------------------------- + + +def test_one_task_per_planned_result_keyed_by_its_address(tmp_path: Path) -> None: + """Built once, per universe, and the stage's output built in the + project's universes because they drive it.""" + graph = plan.build(_project(tmp_path)) + assert sorted(graph.tasks) == [ + "fit#alternative", + "fit#baseline", + "report#alternative", + "report#baseline", + "stage.mask#alternative", + "stage.mask#baseline", + "summary", + ] + assert all(key == task.key for key, task in graph.tasks.items()) + + +def test_a_result_built_per_universe_has_its_place_under_the_universe(tmp_path: Path) -> None: + task = plan.build(_project(tmp_path)).tasks["fit#baseline"] + (place,) = task.places + assert (place.name, place.address) == ("fit", "fit#baseline") + assert place.bytes == "results/baseline/fit.json" + assert place.record == ".astra/records/baseline/fit.json" + + +def test_a_result_built_once_has_no_universe_directory(tmp_path: Path) -> None: + task = plan.build(_project(tmp_path)).tasks["summary"] + (place,) = task.places + assert (place.name, place.address) == ("summary", "summary") + assert place.bytes == "results/summary.csv" + assert place.record == ".astra/records/summary.json" + assert task.decisions == {} + + +def test_the_command_arrives_rendered(tmp_path: Path) -> None: + """A worker is handed a shell command, never a template to fill.""" + task = plan.build(_project(tmp_path)).tasks["fit#alternative"] + assert task.workdir == "" + assert task.command == ( + "python src/fit.py results/summary.csv results/alternative/stage.mask.txt " + "-m nested results/alternative/fit.json" ) - assert {u for u, _ in graph.tasks} == {"baseline", "alternative"} - - -def test_a_decision_that_differs_gives_a_different_definition_version(tmp_path: Path) -> None: - """The whole point of a universe: the same output, made differently, is - a different thing.""" - graph = _build( - _project( - tmp_path, - baseline="id: baseline\ndecisions:\n method: mcmc\n", - alternative="id: alternative\ndecisions:\n method: nested\n", - ) - ) - assert ( - graph.tasks[("baseline", "fit")].definition_version - != graph.tasks[("alternative", "fit")].definition_version - ) - + assert task.decisions == {"method": "nested"} -def test_an_output_that_ignores_a_decision_is_not_moved_by_it(tmp_path: Path) -> None: - """`definition_version` hashes the decisions the output *declares*, so an - unrelated choice does not stale it.""" - a = _build(_project(tmp_path, baseline="id: baseline\ndecisions:\n method: mcmc\n")) - b = _build(_project(tmp_path, baseline="id: baseline\ndecisions:\n method: nested\n")) - key = ("baseline", "report") - assert a.tasks[key].definition_version == b.tasks[key].definition_version +def test_inputs_keep_their_entries_and_name_files_from_the_root(tmp_path: Path) -> None: + task = plan.build(_project(tmp_path)).tasks["fit#baseline"] + assert task.inputs == { + "summary": "results/summary.csv", + "catalog": "data/catalog.csv", + "stage.mask": "results/baseline/stage.mask.txt", + } + assert list(task.inputs) == ["summary", "catalog", "stage.mask"] -def test_an_output_addresses_its_own_file(tmp_path: Path) -> None: - task = _build(_project(tmp_path)).tasks[("baseline", "fit")] - assert task.output_path == tmp_path / "results" / "baseline" / "fit.json" - assert task.manifest_path == tmp_path / "results" / "baseline" / ".fit.manifest.json" - assert "results/baseline/fit.json" in task.recipe +def test_only_inputs_another_task_makes_become_edges(tmp_path: Path) -> None: + graph = plan.build(_project(tmp_path)) + fit = graph.tasks["fit#baseline"] + assert fit.produced_by == {"summary": "summary", "stage.mask": "stage.mask#baseline"} + assert fit.depends_on == ("summary", "stage.mask#baseline") + assert graph.tasks["summary"].produced_by == {} + assert graph.tasks["summary"].depends_on == () -def test_recipe_resources_survive_graph_resolution(tmp_path: Path) -> None: - spec = _SPEC.replace( - "command: python src/fit.py", - "resources: {cpus: 4, memory: 6Gi, time_limit: 1h30m}\n" - " command: python src/fit.py", - ) - graph = _build(_project(tmp_path, spec)) - task = graph.tasks[("baseline", "fit")] - assert task.resources == {"cpus": 4, "memory": "6Gi", "time_limit": "1h30m"} - assert graph.tasks[("baseline", "report")].resources == {} - -@pytest.mark.parametrize( - ("declaration", "expected"), - [ - ("gpus: 1", {"gpus": 1}), - ("disk: 10Gi", {"disk": "10Gi"}), - ("cpus: 0.5", {"cpus": 0.5}), - ("cpus: 0", {"cpus": 0}), - ("memory: null", {"memory": None}), - ], -) -def test_execution_support_does_not_limit_graph_construction( - tmp_path: Path, declaration: str, expected: dict[str, object], -) -> None: - spec = _SPEC.replace( - "command: python src/fit.py", - f"resources: {{{declaration}}}\n command: python src/fit.py", +def test_reading_one_task_twice_is_one_dependency(tmp_path: Path) -> None: + spec = _SPEC.replace("inputs: [fit]", "inputs: [fit, fit#baseline]").replace( + "{inputs.fit}", "{inputs}" ) - task = _build(_project(tmp_path, spec)).tasks[("baseline", "fit")] - assert task.resources == expected + task = plan.build(_project(tmp_path, spec)).tasks["report#baseline"] + assert task.produced_by == {"fit": "fit#baseline", "fit#baseline": "fit#baseline"} + assert task.depends_on == ("fit#baseline",) -def test_a_declared_input_resolves_to_its_source(tmp_path: Path) -> None: - task = _build(_project(tmp_path)).tasks[("baseline", "fit")] - assert task.inputs == {"catalog": tmp_path / "data" / "catalog.fits"} - assert task.produced_by == {} - assert "data/catalog.fits" in task.recipe +def test_tasks_come_after_the_tasks_they_read(tmp_path: Path) -> None: + """Submission walks the graph as given, handing each task its + upstreams' futures — so they must already have been submitted.""" + order = list(plan.build(_project(tmp_path)).tasks) + for key, task in plan.build(_project(tmp_path / "again")).tasks.items(): + assert all(order.index(upstream) < order.index(key) for upstream in task.depends_on) -def test_a_declared_input_outside_the_project_keeps_its_absolute_path( +def test_a_driven_analysis_result_belongs_to_the_project_and_runs_in_its_own_folder( tmp_path: Path, ) -> None: - """A path in a recipe *is* the path on disk, and one outside the tree - has no project-relative spelling to fall back on. It used to be - rendered by `relative_to`, which raised a bare ValueError.""" - outside = tmp_path.parent / "shared" / "catalog.fits" - root = _project(tmp_path, _SPEC.replace("source: data/catalog.fits", f"source: {outside}")) - - task = _build(root).tasks[("baseline", "fit")] - assert task.inputs == {"catalog": outside} - assert str(outside) in task.recipe - - -def test_an_input_another_output_produces_becomes_an_edge(tmp_path: Path) -> None: - task = _build(_project(tmp_path)).tasks[("baseline", "report")] - assert task.produced_by == {"fit": ("baseline", "fit")} - assert task.depends_on == (("baseline", "fit"),) - assert "results/baseline/fit" in task.recipe - - -def test_an_input_nothing_provides_is_an_error(tmp_path: Path) -> None: - """Caught by ASTRA rather than by us: resolution answers what a valid - spec means, so `build` asks whether it is one first — and the error - names the line at fault instead of the run that tripped over it.""" - spec = _SPEC.replace("inputs: [catalog]", "inputs: [missing]").replace( - "{inputs.catalog}", "{inputs.missing}" + """The result is named through the declaration and lands in the + project's `results/`; the recipe is the stage's, so it runs there and + its paths are spelled from there.""" + task = plan.build(_project(tmp_path)).tasks["stage.mask#baseline"] + (place,) = task.places + assert place.name == "stage.mask" + assert place.bytes == "results/baseline/stage.mask.txt" + assert place.record == ".astra/records/baseline/stage.mask.json" + assert task.workdir == "stage" + assert task.command == "python mask.py deep ../results/baseline/stage.mask.txt" + assert task.decisions == {"depth": "deep"} + + +def test_a_result_of_another_analysis_is_keyed_under_its_folder(tmp_path: Path) -> None: + """Read from one of the stage's own universes, the result stays the + stage's — so its address alone would not say whose it is.""" + graph = plan.build(_project(tmp_path, _PINNED)) + task = graph.tasks["stage/mask#v"] + (place,) = task.places + assert place.address == "mask#v" + assert place.bytes == "stage/results/v/mask.txt" + assert place.record == "stage/.astra/records/v/mask.json" + assert task.workdir == "stage" + assert task.command == "python mask.py deep results/v/mask.txt" + + fit = graph.tasks["fit#baseline"] + assert fit.inputs["stage.mask#v"] == "stage/results/v/mask.txt" + assert fit.produced_by["stage.mask#v"] == "stage/mask#v" + + +@pytest.mark.parametrize("spec", [_SPEC, _PINNED]) +def test_a_project_below_its_repository_top_plans_the_same(tmp_path: Path, spec: str) -> None: + """ASTRA measures from the top of the repository; everything lc does + with a path starts at the project root.""" + at_top = plan.build(_project(tmp_path / "top", spec)) + below = plan.build(_project(tmp_path / "nested", spec, below="papers/one")) + assert below == at_top + + +def test_recipe_resources_are_carried_as_declared(tmp_path: Path) -> None: + """Only what was declared: whether this executor can honour it is + decided when executing, not when planning.""" + spec = _SPEC.replace( + "command: python src/summary.py", + "resources: {cpus: 0.5, memory: 6Gi, gpus: 1, time_limit: 1h30m}\n" + " command: python src/summary.py", ) - with pytest.raises(ProjectError, match="does not validate"): - _build(_project(tmp_path, spec)) + graph = plan.build(_project(tmp_path, spec)) + assert graph.tasks["summary"].resources == { + "cpus": 0.5, + "memory": "6Gi", + "gpus": 1, + "time_limit": "1h30m", + } + assert graph.tasks["report#baseline"].resources == {} -def test_a_spec_astra_rejects_never_reaches_a_recipe(tmp_path: Path) -> None: - """The gate exists because the resolver assumes validity: without it an - invalid spec surfaces as a missing decision or an unresolvable input, - blaming the run for a fault in the file.""" - spec = _SPEC.replace('name: demo', "") - with pytest.raises(ProjectError, match="MISSING_ROOT_FIELD"): - _build(_project(tmp_path, spec)) +def test_a_command_reaching_outside_its_analysis_is_noted_not_refused(tmp_path: Path) -> None: + spec = _SPEC.replace("python src/report.py", "python ../src/report.py") + graph = plan.build(_project(tmp_path, spec)) + assert "report#baseline" in graph.tasks + assert any("../src/report.py" in note for note in graph.notes) + assert plan.build(_project(tmp_path / "clean")).notes == () -# ---- ordering, closure, targets -------------------------------------------- +# ---- closure, targets ------------------------------------------------------ -def test_order_puts_dependencies_first(tmp_path: Path) -> None: - order = _build(_project(tmp_path)).order() - assert order.index(("baseline", "fit")) < order.index(("baseline", "report")) +def test_asking_for_an_output_asks_for_what_it_is_made_of(tmp_path: Path) -> None: + graph = plan.build(_project(tmp_path)) + closure = graph.closure(["report#baseline"]) + assert list(closure.tasks) == [ + key + for key in graph.tasks + if key in {"summary", "stage.mask#baseline", "fit#baseline", "report#baseline"} + ] -def test_a_cycle_is_a_clean_error(tmp_path: Path) -> None: - spec = _SPEC.replace(" - id: reexport\n type: metric\n", "") - spec = spec.replace(" inputs: [catalog]", " inputs: [report]") - spec = spec.replace("{inputs.catalog}", "{inputs.report}") - with pytest.raises(ProjectError, match="cycle"): - _build(_project(tmp_path, spec)).order() +def test_a_bare_name_means_every_universe(tmp_path: Path) -> None: + graph = plan.build(_project(tmp_path)) + assert sorted(graph.resolve(["fit"])) == ["fit#alternative", "fit#baseline"] + assert sorted(graph.resolve(["stage.mask"])) == [ + "stage.mask#alternative", + "stage.mask#baseline", + ] + assert graph.resolve(["summary"]) == ["summary"] -def test_asking_for_an_output_asks_for_what_it_is_made_of(tmp_path: Path) -> None: - graph = _build(_project(tmp_path)) - closure = graph.closure(graph.resolve(["report"])) - assert sorted(closure.tasks) == [("baseline", "fit"), ("baseline", "report")] - - -def test_a_bare_target_means_every_universe(tmp_path: Path) -> None: - graph = _build( - _project( - tmp_path, - baseline="id: baseline\ndecisions:\n method: mcmc\n", - alternative="id: alternative\ndecisions:\n method: nested\n", - ) - ) - assert sorted(graph.resolve(["fit"])) == [("alternative", "fit"), ("baseline", "fit")] - - -def test_a_qualified_target_means_exactly_one(tmp_path: Path) -> None: - graph = _build( - _project( - tmp_path, - baseline="id: baseline\ndecisions:\n method: mcmc\n", - alternative="id: alternative\ndecisions:\n method: nested\n", - ) - ) - assert graph.resolve(["baseline/fit"]) == [("baseline", "fit")] +def test_a_key_means_exactly_one(tmp_path: Path) -> None: + graph = plan.build(_project(tmp_path, _PINNED)) + assert graph.resolve(["fit#baseline", "stage/mask#v"]) == ["fit#baseline", "stage/mask#v"] def test_an_unknown_target_is_an_error_not_an_empty_run(tmp_path: Path) -> None: """Quietly making nothing is the least useful thing a build tool can do, so the message lists what there was.""" - graph = _build(_project(tmp_path)) - with pytest.raises(ProjectError, match="no output matches `typo`.*baseline/fit"): + graph = plan.build(_project(tmp_path)) + with pytest.raises(ProjectError, match="no output matches `typo`.*fit#baseline"): graph.resolve(["typo"]) # ---- refusals -------------------------------------------------------------- -def test_two_universes_cannot_claim_one_id(tmp_path: Path) -> None: - """They would materialize into one directory, and the second simply - replaced the first — so a universe went missing with nothing said. The - natural way to reach it is copying a universe file and forgetting to - change the id inside.""" - root = _project( - tmp_path, - baseline="id: baseline\ndecisions:\n method: mcmc\n", - copy="id: baseline\ndecisions:\n method: nested\n", - ) - with pytest.raises(ProjectError, match="both declare the universe `baseline`"): - _build(root) - - -def test_universes_are_told_apart_by_their_id_not_their_filename(tmp_path: Path) -> None: - root = _project( - tmp_path, - baseline="id: baseline\ndecisions:\n method: mcmc\n", - alternative="id: alternative\ndecisions:\n method: nested\n", - ) - assert {u for u, _ in _build(root).tasks} == {"baseline", "alternative"} - - def test_no_spec_is_a_clean_error(tmp_path: Path) -> None: - (tmp_path / "universes").mkdir() with pytest.raises(ProjectError, match="no astra.yaml"): - _build(tmp_path) + plan.build(tmp_path) -def test_no_universe_is_a_clean_error(tmp_path: Path) -> None: - (tmp_path / "astra.yaml").write_text(textwrap.dedent(_SPEC)) - with pytest.raises(ProjectError, match="declares no universe"): - _build(tmp_path) - - -def test_an_output_without_a_format_is_refused_by_name(tmp_path: Path) -> None: - """lc names the file from it, so there is nowhere to write the output. - Every offender at once: a spec is fixed in one pass, not one run per - missing key.""" - root = _project(tmp_path) - (root / "astra.yaml").write_text( - textwrap.dedent(_SPEC).replace(" format: json\n", "").replace(" format: md\n", "") - ) - with pytest.raises(ProjectError, match="format") as raised: - _build(root) - assert "fit" in str(raised.value) and "report" in str(raised.value) - - -# ---- rendering a recipe ---------------------------------------------------- +def test_a_spec_astra_rejects_is_refused_in_astras_words(tmp_path: Path) -> None: + """Every broken rule at once, each placed in its file — a spec is fixed + in one pass, and the fault is the file's rather than the run's.""" + spec = _SPEC.replace("inputs: [catalog]", "inputs: [missing]") + with pytest.raises(ProjectError, match="does not validate") as raised: + plan.build(_project(tmp_path, spec)) + message = str(raised.value) + assert "UNRESOLVED_ADDRESS" in message + assert "UNDECLARED_TEMPLATE_REF" in message + assert "astra.yaml:" in message +def test_an_unparseable_spec_is_a_clean_error(tmp_path: Path) -> None: + with pytest.raises(ProjectError, match="does not validate"): + plan.build(_project(tmp_path, "outputs: [unclosed\n")) +@pytest.mark.parametrize( + ("spec", "fragment"), + [(_SPEC, "`stage.mask#baseline` belongs to"), (_PINNED, "reads `stage.mask#v` from")], +) +def test_an_analysis_declared_with_a_commit_is_refused( + tmp_path: Path, spec: str, fragment: str +) -> None: + """Its results are made from that commit's tree, which is not the tree + a recipe would run in here.""" + root = _project(tmp_path, spec) + _git(root, "commit", "-q", "-m", "stage") + pinned = f"stage: {{path: stage, commit: {_git(root, 'rev-parse', 'HEAD')}}}" + (root / "astra.yaml").write_text(spec.replace("stage: {path: stage}", pinned)) + + with pytest.raises(ProjectError, match="declared with a `commit`") as raised: + plan.build(root) + assert fragment in str(raised.value) + + +def test_a_result_owned_by_a_repository_of_its_own_is_refused(tmp_path: Path) -> None: + """lc commits what it makes, and only in the project's repository.""" + root = _project(tmp_path, _PINNED) + (root / "stage" / ".git").mkdir() + with pytest.raises(ProjectError, match="repository of its own") as raised: + plan.build(root) + assert "stage/mask#v" in str(raised.value) diff --git a/tests/test_project.py b/tests/test_project.py index f05e0eca..f53bb4db 100644 --- a/tests/test_project.py +++ b/tests/test_project.py @@ -22,7 +22,6 @@ #: Every path convergence is responsible for. SCAFFOLD = ( "astra.yaml", - "universes/baseline.yaml", "pyproject.toml", ".python-version", ".gitignore", @@ -110,14 +109,14 @@ def test_converge_writes_the_templates_verbatim(tmp_path: Path) -> None: assert (project / "index.md").read_text() == templates.index_md(title="proj") -def test_a_fresh_spec_is_empty_and_passes_the_gate(tmp_path: Path) -> None: - """The scaffold is an empty analysis, not an example to delete — and - still one ASTRA accepts and lc can plan, so every verb works on a - project nobody has edited yet.""" +def test_a_fresh_spec_passes_the_gate(tmp_path: Path, real_tools: None) -> None: + """The scaffold is the minimal analysis `astra init` writes — one + ASTRA accepts and lc can plan, so every verb works on a project nobody + has edited yet. Real tools, because ASTRA validates against git.""" project = tmp_path / "proj" converge(project) - assert plan.build(project).tasks == {} + assert list(plan.build(project).tasks) == ["result"] def test_converge_scaffolds_no_environment_escalation(tmp_path: Path) -> None: @@ -200,18 +199,15 @@ def test_converge_adopts_an_existing_spec_without_touching_it(tmp_path: Path) -> def test_adoption_adds_no_layout_of_its_own(tmp_path: Path) -> None: - """Neither `src/` nor an empty `universes/` is created beside a - user-authored spec: git cannot track an empty directory, so converging - one would report drift on every fresh clone. Where analysis code lives - is the user's layout (astra-tools#100), and universes are discovered by - glob — a missing directory yields no universes, not an error.""" + """No `src/` is created beside a user-authored spec: git cannot track + an empty directory, so converging one would report drift on every + fresh clone. Where analysis code lives is the user's layout.""" project = tmp_path / "proj" project.mkdir() (project / SPEC_FILENAME).write_text('name: "mine"\n') converge(project) assert not (project / "src").exists() - assert not (project / "universes").exists() assert converge(project).converged diff --git a/tests/test_run.py b/tests/test_run.py index 8412faa5..20335e07 100644 --- a/tests/test_run.py +++ b/tests/test_run.py @@ -6,6 +6,7 @@ from __future__ import annotations +import subprocess import sys from pathlib import Path from typing import Any @@ -21,17 +22,33 @@ ) SPEC = """\ -title: Test +spec_version: "0.1.0" inputs: - - id: local - type: data - source: data/local.csv - - id: remote - type: data - source: https://example.org/nope.csv + local: + path: data/local.csv + absent: + path: data/absent.csv outputs: - - id: best_fit + best_fit: type: metric + format: json + inputs: [local] + recipe: + command: python src/fit.py '{inputs.local}' '{output}' +""" + +STAGE = """\ +spec_version: "0.1.0" +inputs: + sub: + path: data/stage.csv +outputs: + cleaned: + type: data + format: csv + inputs: [sub] + recipe: + command: cp '{inputs.sub}' '{output}' """ @@ -44,6 +61,8 @@ def project(tmp_path: Path) -> Path: (root / "pyproject.toml").write_text('[project]\nname = "proj"\n') (root / "uv.lock").write_text("version = 1\n") (root / ".venv").mkdir() + # ASTRA reads an analysis in its repository; nothing has to be committed. + subprocess.run(["git", "init", "-q", str(root)], check=True) return root @@ -128,54 +147,59 @@ def test_a_declared_project_still_needs_the_lock(project: Path) -> None: assert "missing uv.lock" in str(raised.value) -def test_a_missing_spec_reads_as_an_empty_one(project: Path) -> None: - (project / "astra.yaml").unlink() - assert engine_run.read_spec(project) == {} - assert engine_run.input_paths(project, {}) == [] - - # ---- declared inputs ------------------------------------------------------ def test_declared_file_inputs_become_read_paths(project: Path) -> None: - spec = engine_run.read_spec(project) - assert engine_run.input_paths(project, spec) == [(project / "data" / "local.csv").resolve()] + assert engine_run.input_paths(project) == [(project / "data" / "local.csv").resolve()] -def test_an_input_declared_inside_a_sub_analysis_is_a_read_path_too(project: Path) -> None: +def test_an_input_of_a_declared_analysis_is_a_read_path_too(project: Path) -> None: """A denial the researcher cannot act on is worse than the access it withheld: the file *is* declared, just not at the top of the tree.""" - (project / "data" / "stage.csv").write_text("x\n") - spec = { - "inputs": [{"id": "local", "source": "data/local.csv"}], - "analyses": {"stage": {"inputs": [{"id": "sub", "source": "data/stage.csv"}]}}, - } - assert engine_run.input_paths(project, spec) == [ + (project / "stage" / "data").mkdir(parents=True) + (project / "stage" / "data" / "stage.csv").write_text("x\n") + (project / "stage" / "astra.yaml").write_text(STAGE) + (project / "astra.yaml").write_text(SPEC + "analyses:\n stage:\n path: stage\n") + assert engine_run.input_paths(project) == [ (project / "data" / "local.csv").resolve(), - (project / "data" / "stage.csv").resolve(), + (project / "stage" / "data" / "stage.csv").resolve(), ] -def test_a_source_that_is_not_a_path_is_left_alone(project: Path) -> None: - """ASTRA's `source` is free-form — a URI, a dotted name, a path — so - "is this a path" is answered by whether it resolves to something that - exists, not by parsing.""" - spec = engine_run.read_spec(project) - assert not any("example.org" in str(p) for p in engine_run.input_paths(project, spec)) +def test_a_declared_input_that_is_not_there_is_left_out(project: Path) -> None: + """The read set names files; a declaration whose file is absent has + nothing to grant.""" + assert not any("absent" in str(p) for p in engine_run.input_paths(project)) + +def test_a_missing_spec_declares_no_inputs(project: Path) -> None: + (project / "astra.yaml").unlink() + assert engine_run.input_paths(project) == [] -def test_a_spec_with_no_inputs_is_fine(tmp_path: Path) -> None: + +def test_a_project_outside_a_repository_declares_no_inputs(tmp_path: Path) -> None: + """Best-effort: where ASTRA cannot read the analysis, the probe still + runs, with nothing declared.""" root = tmp_path / "bare" - root.mkdir() - (root / "astra.yaml").write_text("title: Bare\n") - assert engine_run.input_paths(root, engine_run.read_spec(root)) == [] + (root / "data").mkdir(parents=True) + (root / "data" / "local.csv").write_text("a,b\n") + (root / "astra.yaml").write_text(SPEC) + assert engine_run.input_paths(root) == [] + + +def test_a_spec_with_no_inputs_is_fine(project: Path) -> None: + (project / "astra.yaml").write_text('spec_version: "0.1.0"\n') + assert engine_run.input_paths(project) == [] -def test_an_unresolvable_tree_degrades_to_the_top_level_document(project: Path) -> None: - """A probe exists to debug a project, and a spec whose sub-analysis - references are stale is exactly when someone reaches for one.""" - (project / "astra.yaml").write_text(SPEC + "analyses:\n - id: sub\n path: ../gone\n") - assert engine_run.read_spec(project)["inputs"] +def test_an_unreadable_declared_analysis_keeps_the_readable_inputs(project: Path) -> None: + """A probe exists to debug a project, and a spec whose declared + analysis is broken is exactly when someone reaches for one.""" + (project / "stage").mkdir() + (project / "stage" / "astra.yaml").write_text("inputs: [\n") + (project / "astra.yaml").write_text(SPEC + "analyses:\n stage:\n path: stage\n") + assert engine_run.input_paths(project) == [(project / "data" / "local.csv").resolve()] # ---- the uv hop ----------------------------------------------------------- diff --git a/tests/test_sandbox_denial.py b/tests/test_sandbox_denial.py index 700ebab7..ceba55d2 100644 --- a/tests/test_sandbox_denial.py +++ b/tests/test_sandbox_denial.py @@ -79,15 +79,16 @@ def test_an_undeclared_tool_is_named_with_its_remedy(policy: Policy, project: Pa def test_an_undeclared_data_file_gets_the_astra_snippet( policy: Policy, project: Path, tmp_path: Path ) -> None: - """The remedy is copy-pasteable and matches ASTRA's real schema — - inputs carry `source`, not `path`.""" + """The remedy matches ASTRA's real schema — an input is an id mapped + to a `path` inside the project, which is why it says to bring the file + in rather than naming where it was found.""" external = tmp_path / "elsewhere.fits" external.write_text("") stderr = f"PermissionError: [Errno 13] Permission denied: '{external}'\n" joined = "\n".join(denial.explain(stderr, policy, cwd=project)) assert f"cannot read {external}" in joined assert "inputs:" in joined - assert f"source: {external}" in joined + assert "path: data/elsewhere.fits" in joined def test_an_in_tree_write_is_its_own_kind_of_denial(policy: Policy, project: Path) -> None: diff --git a/tests/test_sandbox_policy.py b/tests/test_sandbox_policy.py index 66978d7b..8aaa76a8 100644 --- a/tests/test_sandbox_policy.py +++ b/tests/test_sandbox_policy.py @@ -517,9 +517,8 @@ def test_nonexistent_paths_are_dropped(built: policy_module.Policy) -> None: def test_the_allowlist_version_reaches_the_attestation(built: policy_module.Policy) -> None: - """The exec allowlist is a maintained surface; an output stays - interpretable after it grows only because the version rode along into - the manifest.""" + """The exec allowlist is a maintained surface, so what a backend + attests names the version of it that was in force.""" from lightcone.engine.sandbox.landlock import LandlockBackend from lightcone.engine.sandbox.model import Capability diff --git a/tests/test_sandbox_wrap.py b/tests/test_sandbox_wrap.py index c5245c1e..a88b656b 100644 --- a/tests/test_sandbox_wrap.py +++ b/tests/test_sandbox_wrap.py @@ -119,7 +119,7 @@ def test_the_write_tier_is_restated_after_the_read_only_guard() -> None: readable project tree works there for free. SBPL is last-match-wins, so reproducing that needs the write set to have the final word — or the guard's `(deny file-write* PROJECT)` revokes the output directory - layer 4's recipes write into, and macOS refuses what Linux allows.""" + recipes write into, and macOS refuses what Linux allows.""" project = Path("/tmp/proj") built = Policy( read=(project,), diff --git a/tests/test_serve.py b/tests/test_serve.py index 2e23bc37..dd7ae1c5 100644 --- a/tests/test_serve.py +++ b/tests/test_serve.py @@ -11,6 +11,7 @@ import asyncio import contextlib import http.cookiejar +import io import json import os import signal @@ -24,7 +25,8 @@ import psutil import pytest -from astra.papers.cache import PaperCache +from astra.evidence import papers +from astra.records import Record, RecordedInput, Runner from lightcone.engine import assets from lightcone.engine import serve as engine @@ -33,6 +35,30 @@ REQUIRED_ENV = "LC_UI_TESTS_REQUIRED" _BUNDLE = engine.BUNDLE +SPEC = 'spec_version: "0.1.0"\nlabel: demo\n' + +def _record(address: str) -> Record: + return Record( + address=address, + content_hash="sha256:x", + size=2, + commit="abc123", + environment="urn:lightcone:env:sha256:e", + runner=Runner("lightcone-cli", "0.5.0"), + inputs=(RecordedInput("catalog", "sha256:c", 4),), + started="2026-10-31T08:59:00Z", + finished="2026-10-31T09:00:00Z", + ) + + +RUN = { + "finishedAt": "2026-10-31T09:00:00Z", + "gitRevision": "abc123", + "environment": "urn:lightcone:env:sha256:e", + "cliVersion": "0.5.0", + "inputVersions": {"catalog": "sha256:c"}, +} + @pytest.fixture def state(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Iterator[Path]: @@ -54,10 +80,23 @@ def state(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Iterator[Path]: def project(tmp_path: Path) -> Path: root = tmp_path / "project" root.mkdir() - (root / "astra.yaml").write_text("name: demo\n") + (root / "astra.yaml").write_text(SPEC) return root +def _recorded(project: Path, path: str, address: str) -> None: + """Write the record a run of lc leaves for the result at *path*.""" + (project / path).parent.mkdir(parents=True, exist_ok=True) + (project / path).write_text("{}") + (record,) = ( + place + for place in assets.record_places(path) + if place.endswith(f"/{address.split('#')[0]}.json") + ) + (project / record).parent.mkdir(parents=True, exist_ok=True) + (project / record).write_text(json.dumps(_record(address).to_json())) + + def _request( url: str, method: str = "GET", @@ -134,7 +173,7 @@ def test_a_stopped_servers_cookie_cannot_authenticate_any_new_server( Each request names the first server's host, as the open page's would.""" other = tmp_path / "other" other.mkdir() - (other / "astra.yaml").write_text("name: other\n") + (other / "astra.yaml").write_text(SPEC) first = engine.serve(project) with urllib.request.urlopen(first.url) as page: # the token logs the page in cookie = next( @@ -159,78 +198,82 @@ def accepts(served: engine.Served) -> bool: @pytest.fixture -def papers(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> PaperCache: - """astra's paper cache, where the server's $HOME puts it. The project is - under that $HOME too, as a project usually is — which is where jupyter - starts abbreviating paths to `~`.""" +def home(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """The project is under the server's $HOME, as a project usually is — + which is where jupyter starts abbreviating paths to `~`.""" monkeypatch.setenv("HOME", str(tmp_path)) - return PaperCache(tmp_path / ".cache" / "astra" / "papers") @pytest.fixture -def served(state: Path, project: Path, papers: PaperCache) -> tuple[str, str]: +def served(state: Path, project: Path, home: None) -> tuple[str, str]: """The server's root URL and its token.""" url = urlsplit(engine.serve(project).url) (token,) = parse_qs(url.query)["token"] return f"{url.scheme}://{url.netloc}", token -def test_an_outputs_run_comes_from_its_manifest(served: tuple[str, str], project: Path) -> None: - """Read by lc's own manifest reader, so the page parses no lc format and - the Contents API never has to serve a dotfile.""" +def test_a_results_run_comes_from_its_record(served: tuple[str, str], project: Path) -> None: + """Read by the server, so the page never parses the format and the + Contents API never has to serve a dotfile.""" base, token = served - output = project / "results" / "baseline" / "best_fit.json" - output.parent.mkdir(parents=True) - output.write_text("{}") - manifest = assets.Manifest( - output_id="best_fit", - universe_id="baseline", - recipe="python fit.py", - definition_version="sha256:d", - env_version="sha256:e", - data_version="sha256:x", - decisions={}, - input_versions={"catalog": "sha256:c"}, - git_sha="abc123", - git_remote="", - lc_version="0.5.0", - hermeticity={}, - finished_at="2026-10-31T09:00:00.000Z", - ) - assets.write(assets.manifest_path(output), manifest) + _recorded(project, "results/baseline/best_fit.json", "best_fit#baseline") api = f"{base}/lightcone/api/run?token={token}&path=" status, body = _request(f"{api}results/baseline/best_fit.json") assert status == 200 - assert json.loads(body)["run"] == { - "finishedAt": "2026-10-31T09:00:00.000Z", - "gitRevision": "abc123", - "recipe": "python fit.py", - "environment": "sha256:e", - "cliVersion": "0.5.0", - "inputVersions": {"catalog": "sha256:c"}, - } - (output.parent / "unrecorded.json").write_text("{}") + assert json.loads(body)["run"] == RUN + (project / "results" / "baseline" / "unrecorded.json").write_text("{}") assert json.loads(_request(f"{api}results/baseline/unrecorded.json")[1]) == {"run": None} # Not a file in the project: what the Contents API would refuse is refused. for path in ("results/baseline/absent.json", "../outside.json", "", "results/baseline"): assert _request(f"{api}{path}")[0] in (400, 404), path - # The manifest stays a dotfile to everything else. - hidden = f"{base}/api/contents/results/baseline/.best_fit.manifest.json?token={token}" - assert _request(hidden)[0] == 404 + # The record stays a dotfile to everything else. + hidden = ".astra/records/baseline/best_fit.json" + assert _request(f"{base}/api/contents/{hidden}?token={token}")[0] == 404 + + +@pytest.mark.parametrize( + ("path", "address"), + [ + ("results/best_fit.json", "best_fit"), + ("results/baseline/stage.fit.tar.gz", "stage.fit#baseline"), + ("stage/results/v/fit.json", "fit#v"), + ], +) +def test_a_run_is_found_wherever_astra_places_the_result( + served: tuple[str, str], project: Path, path: str, address: str +) -> None: + """Built once, named through a declared analysis with a dotted format, + or belonging to a declared analysis: the file's path alone finds it.""" + base, token = served + _recorded(project, path, address) + _, body = _request(f"{base}/lightcone/api/run?token={token}&path={path}") + assert json.loads(body)["run"] == RUN + + +def test_a_record_that_does_not_parse_is_no_run(served: tuple[str, str], project: Path) -> None: + base, token = served + _recorded(project, "results/best_fit.json", "best_fit") + (project / ".astra" / "records" / "best_fit.json").write_text("{}") + _, body = _request(f"{base}/lightcone/api/run?token={token}&path=results/best_fit.json") + assert json.loads(body) == {"run": None} def test_an_outputs_state_is_the_engines( - state: Path, papers: PaperCache, analysis: Callable[..., Path] + state: Path, home: None, analysis: Callable[..., Path] ) -> None: """The page shows what `lc status` would print, from the same call — it holds no classification of its own.""" from lightcone.engine import materialize - root = analysis( - 'version: "0.0.13"\nname: analysis\ninputs: []\ndecisions: {}\n' - "outputs:\n - id: fit\n type: metric\n format: json\n" - " recipe:\n command: 'echo 1 > {output}'\n" - ) + root = analysis("""\ + spec_version: "0.1.0" + outputs: + fit: + type: metric + format: json + recipe: + command: echo 1 > '{output}' + """) url = urlsplit(engine.serve(root).url) status, body = _request(f"{url.scheme}://{url.netloc}/lightcone/api/status?{url.query}") assert status == 200 @@ -239,7 +282,7 @@ def test_an_outputs_state_is_the_engines( "outputs": {o.output: {"state": o.status, "detail": o.why} for o in report.outputs}, "warnings": report.warnings, } - assert json.loads(body)["outputs"]["baseline/fit"]["state"] == "stale" + assert json.loads(body)["outputs"]["fit"]["state"] == "stale" def test_a_project_the_engine_cannot_read_has_no_state(served: tuple[str, str]) -> None: @@ -250,16 +293,21 @@ def test_a_project_the_engine_cannot_read_has_no_state(served: tuple[str, str]) def test_a_run_record_beyond_a_symlink_is_not_read( served: tuple[str, str], project: Path, tmp_path: Path ) -> None: - """The contents manager follows symlinks, so the sidecar the route would - read is checked where it really is.""" + """The contents manager follows symlinks, so the record the route + would read is checked where it really is.""" base, token = served + api = f"{base}/lightcone/api/run?token={token}&path=results/fit.json" + _recorded(project, "results/fit.json", "fit") + assert json.loads(_request(api)[1])["run"] == RUN elsewhere = tmp_path / "elsewhere" elsewhere.mkdir() - (elsewhere / "fit.json").write_text("{}") - (elsewhere / ".fit.manifest.json").write_text("{}") - (project / "results").symlink_to(elsewhere) - status, _ = _request(f"{base}/lightcone/api/run?token={token}&path=results/fit.json") - assert status == 404 + (project / ".astra").rename(elsewhere / ".astra") + (project / ".astra").symlink_to(elsewhere / ".astra") + assert json.loads(_request(api)[1]) == {"run": None} + # And the result itself, as for every file the server sends. + (project / "results").rename(elsewhere / "results") + (project / "results").symlink_to(elsewhere / "results") + assert _request(api)[0] == 404 def test_the_token_reads_and_runs_nothing(served: tuple[str, str], project: Path) -> None: @@ -275,7 +323,7 @@ def test_the_token_reads_and_runs_nothing(served: tuple[str, str], project: Path ("PUT", "/api/config/notebook"), ): assert _request(f"{base}{route}", method, token)[0] == 404, route - overwrite = {"type": "file", "format": "text", "content": "name: overwritten\n"} + overwrite = {"type": "file", "format": "text", "content": "label: overwritten\n"} for method, route, body in ( ("PUT", "/api/contents/astra.yaml", overwrite), ("PUT", "/api/contents/new.yaml", overwrite), @@ -330,26 +378,26 @@ def test_reading_a_notebook_leaves_jupyters_data_directory_alone( assert not list(data.glob("*")) -def test_a_cited_paper_comes_from_astras_cache( - served: tuple[str, str], papers: PaperCache -) -> None: +def test_a_cited_paper_comes_from_astras_cache(served: tuple[str, str]) -> None: + from pypdf import PdfWriter + base, token = served - papers.add("10.1093/MNRAS/stv1198", b"%PDF-1.4 gd1", title="Gaia", authors=["A. B.", "C. D."]) - # The page sends DOIs as the SDK normalizes them; astra stored this one as written. + writer = PdfWriter() + writer.add_blank_page(width=72, height=72) + writer.write(pdf := io.BytesIO()) + papers.add("10.1093/MNRAS/stv1198", None, data=pdf.getvalue()) + # The page sends DOIs as the SDK normalizes them; this one was added as written. query = f"doi=10.1093/mnras/stv1198&doi=10.1/absent&token={token}" status, body = _request(f"{base}/lightcone/api/papers?{query}") assert status == 200 found = json.loads(body)["papers"] assert found.keys() == {"10.1093/mnras/stv1198"} - paper = found["10.1093/mnras/stv1198"] - assert (paper["title"], paper["authors"]) == ("Gaia", "A. B., C. D.") - # Served by Jupyter's own file handler: ranges for pdf.js. - request = urllib.request.Request( - f"{base}{paper['pdfUrl']}?token={token}", headers={"Range": "bytes=0-3"} - ) - with urllib.request.urlopen(request) as pdf: - assert (pdf.status, pdf.headers["Content-Type"]) == (206, "application/pdf") - assert pdf.read() == b"%PDF" + with urllib.request.urlopen( + f"{base}{found['10.1093/mnras/stv1198']['pdfUrl']}&token={token}" + ) as response: + assert (response.status, response.headers["Content-Type"]) == (200, "application/pdf") + assert response.read() == pdf.getvalue() + assert _request(f"{base}/lightcone/api/paper?doi=10.1/absent&token={token}")[0] == 404 def test_every_route_needs_the_token(served: tuple[str, str]) -> None: @@ -359,9 +407,9 @@ def test_every_route_needs_the_token(served: tuple[str, str]) -> None: routes = ( "/lightcone", "/lightcone/api/papers?doi=10.1/x", + "/lightcone/api/paper?doi=10.1/x", "/lightcone/api/run?path=x", "/lightcone/api/status", - "/lightcone/papers/x/paper.pdf", ) for route in routes: assert _request(f"{base}{route}")[0] in (302, 403), route @@ -392,7 +440,7 @@ async def subscribe_then_edit() -> dict[str, object]: ) await socket.write_message(json.dumps({"type": "client_opened", "path": "astra.yaml"})) await asyncio.sleep(0.5) # the watcher starts on subscription - (project / "astra.yaml").write_text("name: edited\n") + (project / "astra.yaml").write_text(SPEC.replace("demo", "edited")) message = await asyncio.wait_for(socket.read_message(), timeout=10) socket.close() assert isinstance(message, str) @@ -560,15 +608,19 @@ def test_metadata_hash_requests_never_read_artifact_bytes(project: Path) -> None from lightcone.app.files import ProjectContents - artifact = project / "large.json" + artifact = project / "results" / "large.json" + artifact.parent.mkdir() with artifact.open("wb") as file: file.truncate(3 * 1024**3) manager = ProjectContents(root_dir=str(project)) with patch.object(manager, "open", side_effect=AssertionError("artifact was read")): - model = asyncio.run(manager.get("large.json", content=False, require_hash=True)) + model = asyncio.run(manager.get("results/large.json", content=False, require_hash=True)) assert model["size"] == 3 * 1024**3 assert model["hash"] is None - assert model["watchPaths"] == ["large.json", ".large.manifest.json"] + assert model["watchPaths"] == ["results/large.json", ".astra/records/large.json"] + # A file at no result's place has no run to watch for. + (project / "notes.json").write_text("{}") + assert asyncio.run(manager.get("notes.json", content=False))["watchPaths"] == ["notes.json"] def test_confined_open_refuses_a_directory_swap(project: Path, tmp_path: Path) -> None: @@ -616,16 +668,18 @@ def test_late_run_record_is_announced_without_changing_the_artifact( ) -> None: from tornado.websocket import websocket_connect - artifact = project / "metric.json" + artifact = project / "results" / "metric.json" + artifact.parent.mkdir() artifact.write_text("{}") + (project / ".astra" / "records").mkdir(parents=True) before = artifact.stat() url = urlsplit(engine.serve(project).url) base = f"{url.scheme}://{url.netloc}" - _, body = _request(f"{base}/api/contents/metric.json?{url.query}") - watch_path = json.loads(body)["watchPaths"][1] - assert watch_path == assets.manifest_path(artifact).relative_to(project).as_posix() + _, body = _request(f"{base}/api/contents/results/metric.json?{url.query}") + watch_paths = json.loads(body)["watchPaths"][1:] + assert watch_paths == [".astra/records/metric.json"] - async def write_run() -> dict[str, object]: + async def write_run(watch_path: str) -> dict[str, object]: socket = await websocket_connect(f"ws://{url.netloc}/api/live-content/ws?{url.query}") try: await socket.write_message(json.dumps({"type": "client_opened", "path": watch_path})) @@ -636,26 +690,28 @@ async def write_run() -> dict[str, object]: finally: socket.close() - update = asyncio.run(write_run()) - assert update == {"type": "server_update", "path": watch_path, "hash": None} + for watch_path in watch_paths: + update = asyncio.run(write_run(watch_path)) + assert update == {"type": "server_update", "path": watch_path, "hash": None} + assert _request(f"{base}/api/contents/{watch_path}?{url.query}")[0] == 404 assert artifact.stat().st_mtime_ns == before.st_mtime_ns - assert _request(f"{base}/api/contents/{watch_path}?{url.query}")[0] == 404 def test_symlink_metadata_describes_the_target(project: Path) -> None: from lightcone.app.files import ProjectContents - target = project / "target.json" + target = project / "results" / "target.json" + target.parent.mkdir() with target.open("wb") as file: file.truncate(2_500_000) - (project / "alias.json").symlink_to(target) + (project / "results" / "alias.json").symlink_to(target) manager = ProjectContents(root_dir=str(project)) - before = asyncio.run(manager.get("alias.json", content=False)) - assert before["path"] == "alias.json" + before = asyncio.run(manager.get("results/alias.json", content=False)) + assert before["path"] == "results/alias.json" assert before["size"] == 2_500_000 - assert before["watchPaths"] == ["target.json", ".alias.manifest.json"] + assert before["watchPaths"] == ["results/target.json", ".astra/records/alias.json"] os.utime(target, (1_700_000_000, 1_700_000_000)) - after = asyncio.run(manager.get("alias.json", content=False)) + after = asyncio.run(manager.get("results/alias.json", content=False)) assert after["last_modified"] != before["last_modified"] assert after["last_modified"].timestamp() == 1_700_000_000 diff --git a/tests/test_templates.py b/tests/test_templates.py index f335a2fb..e879d943 100644 --- a/tests/test_templates.py +++ b/tests/test_templates.py @@ -123,7 +123,7 @@ def test_gitattributes_routes_content_to_the_annex_and_everything_else_to_git() assert entries[0] == "* annex.largefiles=nothing" assert "results/** annex.largefiles=anything" in entries assert "data/** annex.largefiles=anything" in entries - assert "results/**/.*.manifest.json annex.largefiles=nothing" in entries + assert "**/results/** annex.largefiles=anything" in entries def test_gitattributes_exceptions_come_after_the_default() -> None: diff --git a/tests/test_view_browser.py b/tests/test_view_browser.py index ab9707a6..8e2ddac9 100644 --- a/tests/test_view_browser.py +++ b/tests/test_view_browser.py @@ -6,17 +6,18 @@ from __future__ import annotations +import json import os from collections.abc import Iterator from pathlib import Path import pytest +from astra.universes import Place from playwright.sync_api import Browser, Error, expect, sync_playwright -from test_serve import _gate +from test_serve import _gate, _record from test_serve import project as project from test_serve import state as state -from lightcone.engine import assets from lightcone.engine import serve as engine @@ -38,20 +39,24 @@ def test_open_dialog_follows_artifacts_and_late_provenance( browser: Browser, state: Path, project: Path ) -> None: (project / "astra.yaml").write_text("""\ -version: "0.0.14" -name: Viewer regression +spec_version: "0.1.0" +label: Viewer regression description: Live viewer regression -inputs: [] -decisions: {} +decisions: + binning: + options: {coarse: {}, fine: {}} +universes: + baseline: + decisions: {binning: coarse} outputs: - - id: metric + metric: type: metric format: json description: A changing metric + decisions: [binning] + recipe: + command: python metric.py '{decisions.binning}' '{output}' """) - universes = project / "universes" - universes.mkdir() - (universes / "baseline.yaml").write_text("id: baseline\ndecisions: {}\n") # A project-local link must follow edits to the target, too. spec = project / "specs" / "main.yaml" spec.parent.mkdir() @@ -97,23 +102,10 @@ def test_open_dialog_follows_artifacts_and_late_provenance( expect(page.get_by_text("second-value", exact=True).first).to_be_visible() expect(provenance).to_contain_text("No recorded run yet.") before = artifact.stat().st_mtime_ns - manifest = assets.Manifest( - output_id="metric", - universe_id="baseline", - recipe="python metric.py", - definition_version="sha256:d", - env_version="sha256:e", - data_version="sha256:x", - decisions={}, - input_versions={}, - git_sha="abc12345", - git_remote="", - lc_version="0.5.0", - hermeticity={}, - finished_at="2026-10-31T09:00:00.000Z", - ) - assets.write(assets.manifest_path(artifact), manifest) - expect(provenance).to_contain_text("abc12345") + place = Place("metric", "metric#baseline", ".astra/records/baseline/metric.json", wanted) + (project / place.record).parent.mkdir(parents=True) + (project / place.record).write_text(json.dumps(_record(place.address).to_json())) + expect(provenance).to_contain_text("abc123") assert artifact.stat().st_mtime_ns == before spec.write_text(spec.read_text().replace("Viewer regression", "Updated analysis")) expect(page).to_have_title("Updated analysis") diff --git a/tests/test_worker.py b/tests/test_worker.py index 8b392c2a..626f2686 100644 --- a/tests/test_worker.py +++ b/tests/test_worker.py @@ -3,7 +3,7 @@ This file runs real recipes through the real boundary, against a real project, because that is the only way to answer what it asks: whether the environment gates hold, whether the output directory is really reset, -whether the manifest agrees with the bytes beside it, and whether a +whether the record agrees with the bytes it describes, and whether a recipe can reach something it was not given. Following `test_sandbox_enforcement.py`'s rule, every denial assertion is @@ -13,35 +13,41 @@ from __future__ import annotations +import json +import shutil import subprocess import sys from collections.abc import Callable -from dataclasses import replace from pathlib import Path +from typing import Any import pytest +from astra.hashing import digest_bytes, digest_file -from lightcone.engine import assets, container, identity, plan, worker +from lightcone.engine import assets, container, dataset, identity, plan, sandbox, worker from lightcone.engine.sandbox import Unavailable from lightcone.engine.worker import TaskResult _SPEC = """ -version: "0.0.13" -name: analysis +spec_version: "0.1.0" -inputs: - - id: catalog - type: data - source: data/catalog.fits +decisions: + word: + options: {one: {}, two: {}} + +universes: + baseline: + decisions: {word: one} outputs: - - id: first + first: type: metric format: txt + decisions: [word] recipe: - command: echo one > {output} + command: echo {decisions.word} > {output} - - id: second + second: type: report format: txt inputs: [first] @@ -49,48 +55,74 @@ command: cat {inputs.first} > {output} """ +#: Results built once, the first of them from a tracked input. +_READS = """ +spec_version: "0.1.0" -@pytest.fixture -def root(analysis: Callable[..., Path]) -> Path: - return analysis(_SPEC) +inputs: + catalog: {path: data/catalog.txt} +outputs: + fit: + type: metric + format: txt + inputs: [catalog] + recipe: + command: cat {inputs.catalog} > {output} -def _task(root: Path, output_id: str) -> plan.Task: - graph = plan.build(root) - return graph.tasks[("baseline", output_id)] + summary: + type: report + format: txt + inputs: [fit] + recipe: + command: wc -c < {inputs.fit} > {output} +""" +_FIRST = "results/baseline/first.txt" +_SECOND = "results/baseline/second.txt" +_FIRST_RECORD = ".astra/records/baseline/first.json" +_SECOND_RECORD = ".astra/records/baseline/second.json" -#: What the driver reads once and hands to every task. -_HEAD = ("0123456789abcdef", "https://example/analysis.git") +@pytest.fixture +def root(analysis: Callable[..., Path]) -> Path: + return analysis(_SPEC) -def _runtime(root: Path) -> container.Runtime: - """The direct-mode runtime the driver would resolve for these projects.""" - return container.runtime_for_run(root, build=False) +def _task(root: Path, key: str) -> plan.Task: + return plan.build(root).tasks[key] -def _context(root: Path, env_version: str | None = None) -> worker.RunContext: - """The driver-resolved facts a real run would hand down.""" + +def _context(root: Path, environment: str | None = None) -> worker.RunContext: + """The driver-resolved facts a real run would hand down. + + The commit is the repository's own HEAD: a record names the commit its + result is compared against, so an invented one would leave every + result stale. + """ return worker.RunContext( - env_version=env_version if env_version is not None else identity.env_version(root), - head=_HEAD, - versions=assets.Versions(), - runtime=_runtime(root), - uv_version="0.0.0-test", + environment=environment if environment is not None else identity.environment(root), + commit=dataset.head(root)[0], + runtime=container.runtime_for_run(root, build=False), ) -def _make( - root: Path, output_id: str, *upstream: TaskResult, refresh: bool = False -) -> TaskResult: - """Run one task the way Dask would, handed its upstream results.""" - return worker.materialize( - root, _task(root, output_id), _context(root), refresh, None, *upstream - ) +def _make(root: Path, key: str, *upstream: TaskResult, refresh: bool = False) -> TaskResult: + """Run one task the way Dask would, classified beforehand as the driver + does and handed its upstream results.""" + task = _task(root, key) + context = _context(root) + decided = worker.decide(assets.check(root)[task.places[0].record], context.environment, None) + return worker.materialize(root, task, context, refresh, decided, None, *upstream) -def _age(root: Path, output_id: str) -> None: - """Rewrite an output's manifest to name an environment that is not this +def _record(root: Path, path: str) -> dict[str, Any]: + document: dict[str, Any] = json.loads((root / path).read_text()) + return document + + +def _age(root: Path, record: str) -> None: + """Rewrite a result's record to name an environment that is not this one — the shape a project takes when `uv add` rewrites the lock after a result was made. @@ -98,152 +130,261 @@ def _age(root: Path, output_id: str) -> None: the recipe actually runs under has to stay the real one, or the mid-run gate refuses the execution before any of this is exercised. """ - sidecar = root / "results/baseline" / f".{output_id}.manifest.json" - manifest = assets.read(sidecar) - assert manifest is not None - assets.write(sidecar, replace(manifest, env_version="sha256:an-earlier-environment")) + document = json.loads((root / record).read_text()) + document["generated_by"]["environment"]["iri"] = "urn:lightcone:env:sha256:an-earlier-one" + (root / record).write_text(json.dumps(document)) # ---- executing a recipe ---------------------------------------------------- def test_a_recipe_runs_and_its_output_is_recorded(root: Path) -> None: - result = _make(root, "first") + result = _make(root, "first#baseline") assert result.status == "ok" - assert (root / "results/baseline/first.txt").read_text() == "one\n" - assert result.data_version == assets.data_version(root / "results/baseline/first.txt") - - -def test_the_manifest_is_complete_before_anything_is_saved(root: Path) -> None: - """The driver commits the directory and its manifest as one commit, so - a manifest that described the bytes only after the save could never be - in that commit.""" - _make(root, "first") - - manifest = assets.read(root / "results/baseline/.first.manifest.json") - assert manifest is not None - assert manifest.data_version == assets.data_version(root / "results/baseline/first.txt") - assert manifest.definition_version == _task(root, "first").definition_version - assert manifest.env_version == identity.env_version(root) - assert manifest.git_sha == _HEAD[0] and manifest.git_remote == _HEAD[1] - assert manifest.hermeticity["mechanism"] + assert (root / _FIRST).read_text() == "one\n" + assert (result.content_hash, result.size) == (digest_bytes(b"one\n").content_hash, 4) + + +def test_the_record_says_what_the_run_was(root: Path) -> None: + """The record is what the result is checked against from then on, so + it has to name this result, these bytes, the commit the run started + from and the environment it ran under — and be one ASTRA accepts.""" + result = _make(root, "first#baseline") + + record = _record(root, _FIRST_RECORD) + run = record["generated_by"] + assert record["address"] == "first#baseline" + assert (record["content_hash"], record["size"]) == (result.content_hash, result.size) + assert run["commit"] == dataset.head(root)[0] + assert run["environment"] == {"iri": identity.environment(root)} # The engine's version is attestation, not identity: with lc outside # the project's lock, this field is the record of which engine ran. - assert manifest.lc_version == worker.lc_version() - # And so is the uv that converged the environment: probed once by the - # driver, handed down, never a rebuild signal. - assert manifest.uv_version == "0.0.0-test" + assert run["runner"] == {"id": "lightcone-cli", "version": worker.lc_version()} + assert run["started"] <= run["finished"] + assert "inputs" not in run + assert assets.check(root)[_FIRST_RECORD][0].record_status == "unchanged" + + +def test_a_result_built_once_has_no_universe_directory(analysis: Callable[..., Path]) -> None: + """And it reads a declared input, which lives in the annex. With + `filter=annex` the working tree holds the real bytes, so the project + read root covers them — worth pinning, since it is what every recipe + with an input depends on.""" + root = analysis(_READS, files={"data/catalog.txt": "measured\n"}) + result = _make(root, "fit") + + assert result.status == "ok", result.reason + assert (root / "results/fit.txt").read_text() == "measured\n" + record = _record(root, ".astra/records/fit.json") + assert record["address"] == "fit" + read = digest_bytes(b"measured\n") + assert record["generated_by"]["inputs"] == [ + {"entry": "catalog", "content_hash": read.content_hash, "size": read.size} + ] -def test_the_recipe_runs_under_the_boundary(root: Path) -> None: - """Whatever the host can enforce, the manifest records what it was — - never what the mechanism matrix says it should have been.""" - from lightcone.engine import sandbox - _make(root, "first") +def test_a_driven_analysis_is_built_in_its_own_folder_for_the_project( + analysis: Callable[..., Path], +) -> None: + """The result belongs to the project — its bytes and record land + under the project's own `results/` and `.astra/` — while the + recipe runs where its analysis's relative paths mean something.""" + spec = """ + spec_version: "0.1.0" - manifest = assets.read(root / "results/baseline/.first.manifest.json") - assert manifest is not None - assert manifest.hermeticity["mechanism"] == sandbox.detect().capability.kind + analyses: + stage: {path: stage} + + universes: + u1: + decisions: {stage.dec: a} + + outputs: + report: + type: report + format: txt + inputs: [stage.out] + recipe: + command: cat {inputs.stage.out} > {output} + """ + stage = """ + spec_version: "0.1.0" + + decisions: + dec: + options: {a: {}, b: {}} + + universes: + v1: + decisions: {dec: b} + + outputs: + out: + type: metric + format: txt + decisions: [dec] + recipe: + command: echo {decisions.dec} $(cat here.txt) > {output} + """ + root = analysis(spec, files={"stage/astra.yaml": stage, "stage/here.txt": "in stage\n"}) + + made = _make(root, "stage.out#u1") + report = _make(root, "report#u1", made) + + assert made.status == "ok", made.reason + assert _task(root, "stage.out#u1").workdir == "stage" + assert (root / "results/u1/stage.out.txt").read_text() == "a in stage\n" + assert _record(root, ".astra/records/u1/stage.out.json")["address"] == "stage.out#u1" + assert not (root / "stage/results").exists() + assert report.status == "ok", report.reason + assert (root / "results/u1/report.txt").read_text() == "a in stage\n" # ---- deciding whether to run at all ---------------------------------------- def test_an_unchanged_output_is_current(root: Path) -> None: - made = _make(root, "first") - again = _make(root, "first") + made = _make(root, "first#baseline") + again = _make(root, "first#baseline") assert again.status == "current" - assert again.data_version == made.data_version + assert (again.content_hash, again.size) == (made.content_hash, made.size) def test_a_skip_returns_the_recorded_digest_rather_than_rehashing(root: Path) -> None: """On a clone that has fetched no annex content the files are dangling symlinks, so a recompute would quietly report a different output.""" - _make(root, "first") - output = root / "results/baseline/first.txt" - manifest = assets.read(assets.manifest_path(output)) - assert manifest is not None - output.unlink() + made = _make(root, "first#baseline") + (root / _FIRST).unlink() + + again = _make(root, "first#baseline") - assert _make(root, "first").data_version == manifest.data_version + assert again.status == "current" + assert again.content_hash == made.content_hash def test_a_moved_environment_leaves_the_output_alone(root: Path) -> None: """The change this layer exists for. The recipe and the decisions still define exactly this output, so it is reported and kept — remaking it would spend the compute that a rewritten `uv.lock` never justified.""" - made = _make(root, "first") - (root / "results/baseline/first.txt").write_text("untouched\n") - _age(root, "first") + made = _make(root, "first#baseline") + (root / _FIRST).write_text("untouched\n") + _age(root, _FIRST_RECORD) - again = _make(root, "first") + again = _make(root, "first#baseline") assert again.status == "behind" assert "earlier environment" in again.reason - assert again.data_version == made.data_version - assert (root / "results/baseline/first.txt").read_text() == "untouched\n" + assert again.content_hash == made.content_hash + assert (root / _FIRST).read_text() == "untouched\n" def test_refresh_remakes_what_is_only_behind(root: Path) -> None: """And the recipe really runs: the file the previous assertion left in place is overwritten, so this cannot pass by skipping too.""" - _make(root, "first") - (root / "results/baseline/first.txt").write_text("untouched\n") - _age(root, "first") + _make(root, "first#baseline") + (root / _FIRST).write_text("untouched\n") + _age(root, _FIRST_RECORD) - again = _make(root, "first", refresh=True) + again = _make(root, "first#baseline", refresh=True) assert again.status == "ok" - assert (root / "results/baseline/first.txt").read_text() == "one\n" + assert (root / _FIRST).read_text() == "one\n" def test_refresh_does_not_remake_what_is_current(root: Path) -> None: """`--refresh` widens the run by one state, not to everything. Without this it would be a rebuild-the-world flag wearing another name.""" - _make(root, "first") + _make(root, "first#baseline") + + assert _make(root, "first#baseline", refresh=True).status == "current" - assert _make(root, "first", refresh=True).status == "current" + +def _reads_first(result: TaskResult) -> list[dict[str, object]]: + """What a record of `second` holds when it read *result*.""" + return [{"entry": "first", "content_hash": result.content_hash, "size": result.size}] def test_a_behind_upstream_still_feeds_its_dependents(root: Path) -> None: """`behind` says the environment moved, not that the bytes are wrong — so a dependent proceeds on them rather than reporting blocked.""" - first = _make(root, "first") - _age(root, "first") - behind = _make(root, "first") + first = _make(root, "first#baseline") + _age(root, _FIRST_RECORD) + behind = _make(root, "first#baseline") assert behind.status == "behind" - second = _make(root, "second", behind) + second = _make(root, "second#baseline", behind) assert second.status == "ok" - manifest = assets.read(root / "results/baseline/.second.manifest.json") - assert manifest is not None - assert manifest.input_versions == {"first": first.data_version} + assert _record(root, _SECOND_RECORD)["generated_by"]["inputs"] == _reads_first(first) def test_a_task_whose_upstream_did_not_finish_is_blocked(root: Path) -> None: """Blocked without running: an exception would make Dask propagate to every dependent and stop 'who actually failed' being answerable.""" - upstream = TaskResult(("baseline", "first"), "failed", reason="boom") + upstream = TaskResult("first#baseline", "failed", reason="boom") - result = _make(root, "second", upstream) + result = _make(root, "second#baseline", upstream) assert result.status == "blocked" - assert "baseline/first" in result.reason - assert not (root / "results/baseline/second.txt").exists() + assert "first#baseline" in result.reason + assert not (root / _SECOND).exists() def test_a_downstream_task_takes_its_upstreams_answer(root: Path) -> None: - first = _make(root, "first") - second = _make(root, "second", first) + """What the upstream task reported, never a rehash of the file: where + the upstream was left alone on a clone without its content, the file + is a pointer, and its hash is not the result's.""" + first = _make(root, "first#baseline") + (root / _FIRST).write_text("not the bytes the upstream task reported\n") + + second = _make(root, "second#baseline", first) assert second.status == "ok" - manifest = assets.read(root / "results/baseline/.second.manifest.json") - assert manifest is not None - assert manifest.input_versions == {"first": first.data_version} + assert first.content_hash != digest_file(root / _FIRST).content_hash + assert _record(root, _SECOND_RECORD)["generated_by"]["inputs"] == _reads_first(first) + + +# ---- downstream of a task that may be remade ------------------------------- + + +def _ask(root: Path, key: str, *upstream: TaskResult) -> TaskResult: + """Run one task the driver could not classify beforehand, because + something it reads was about to be remade.""" + return worker.materialize(root, _task(root, key), _context(root), False, None, None, *upstream) + + +def test_a_byte_identical_remake_leaves_its_dependents_current(root: Path) -> None: + """Where the cascade stops: the dependent's record still names the + bytes it read, so there is nothing to make again.""" + second = _make(root, "second#baseline", _make(root, "first#baseline")) + (root / _SECOND).write_text("untouched\n") + remade = worker.execute(root, _task(root, "first#baseline"), {}, _context(root)) + assert remade.status == "ok" + + again = _ask(root, "second#baseline", remade) + + assert again.status == "current" + assert again.content_hash == second.content_hash + assert (root / _SECOND).read_text() == "untouched\n" + + +def test_a_remake_with_other_bytes_remakes_its_dependents( + analysis: Callable[..., Path], +) -> None: + root = analysis(_READS, files={"data/catalog.txt": "measured\n"}) + _make(root, "summary", _make(root, "fit")) + assert (root / "results/summary.txt").read_text().strip() == "9" + (root / "data/catalog.txt").write_text("measured again\n") + remade = _make(root, "fit") + assert remade.status == "ok" + + again = _ask(root, "summary", remade) + + assert again.status == "ok" + assert (root / "results/summary.txt").read_text().strip() == "15" # ---- the output directory the recipe owns ---------------------------------- @@ -253,66 +394,68 @@ def test_a_payload_from_an_earlier_format_does_not_survive_a_rebuild(root: Path) """The reset takes every file named after the output, not just the one the spec declares today — so re-declaring the serialization leaves no orphan behind for the publication view to trip over.""" - _make(root, "first") + _make(root, "first#baseline") earlier = root / "results/baseline/first.json" earlier.write_text("from a run that declared another format\n") - worker.execute(root, _task(root, "first"), {}, _context(root)) + worker.execute(root, _task(root, "first#baseline"), {}, _context(root)) assert not earlier.exists() - assert (root / "results/baseline/first.txt").exists() + assert (root / _FIRST).exists() def test_a_sibling_output_is_left_alone_by_the_reset(root: Path) -> None: - """Outputs share a directory now, and under Dask they are written - concurrently — so the reset may only ever touch what its own id + """Outputs share a directory, and under Dask they are written + concurrently — so the reset may only ever touch what its own name names.""" - _make(root, "first") - sibling = root / "results/baseline/second.txt" - sibling.parent.mkdir(parents=True, exist_ok=True) + _make(root, "first#baseline") + sibling = root / _SECOND sibling.write_text("a neighbour's bytes\n") - worker.execute(root, _task(root, "first"), {}, _context(root)) + worker.execute(root, _task(root, "first#baseline"), {}, _context(root)) assert sibling.read_text() == "a neighbour's bytes\n" -def test_a_failing_recipe_records_no_manifest(root: Path) -> None: - spec = _SPEC.replace("echo one > {output}", "echo one > {output} && false") - (root / "astra.yaml").write_text(spec) +def _recorded(root: Path) -> bool: + """Whether anything says `first` was made.""" + return (root / _FIRST_RECORD).exists() + - result = _make(root, "first") +def test_a_failing_recipe_records_nothing(root: Path) -> None: + (root / "astra.yaml").write_text(_SPEC.replace("> {output}", "> {output} && false", 1)) + + result = _make(root, "first#baseline") assert result.status == "failed" assert "exited 1" in result.reason - assert assets.read(root / "results/baseline/.first.manifest.json") is None + assert not _recorded(root) def test_a_recipe_that_writes_nothing_fails(root: Path) -> None: """Exit 0 is not evidence of an output. Without this the run would - commit a manifest for a file that is not there.""" - (root / "astra.yaml").write_text( - _SPEC.replace("echo one > {output}", "echo nowhere > /dev/null") - ) + commit a record for a file that is not there.""" + (root / "astra.yaml").write_text(_SPEC.replace("> {output}", "> /dev/null", 1)) - result = _make(root, "first") + result = _make(root, "first#baseline") assert result.status == "failed" assert "left nothing" in result.reason - assert assets.read(root / "results/baseline/.first.manifest.json") is None + assert not _recorded(root) def test_a_recipe_that_writes_a_directory_fails(root: Path) -> None: - """`data_version` hashes a directory perfectly happily, so `mkdir` on - the output — which is what every recipe written for the old layout - does — would otherwise commit a well-formed digest of something that - is not the output at all.""" - (root / "astra.yaml").write_text(_SPEC.replace("echo one > {output}", "mkdir -p {output}")) + """A result is one file, and a record of a directory would describe + something that is not the output at all.""" + (root / "astra.yaml").write_text( + _SPEC.replace("echo {decisions.word} > {output}", "mkdir -p {output}") + ) - result = _make(root, "first") + result = _make(root, "first#baseline") assert result.status == "failed" assert "a directory" in result.reason + assert not _recorded(root) def test_an_output_that_cannot_be_recorded_fails_rather_than_raises( @@ -321,54 +464,89 @@ def test_an_output_that_cannot_be_recorded_fails_rather_than_raises( """Recording is fallible too, and a raise here reaches Dask, which re-raises in the driver and takes down every other task in flight — where reporting one failure and letting the rest finish is the whole - point of owning the loop. Exercised without a mechanism, because that - is the host where a recipe really can delete what it was given.""" + point of owning the loop.""" def refuse(*args: object, **kwargs: object) -> None: raise OSError("no space left on device") - monkeypatch.setattr(assets, "write", refuse) + monkeypatch.setattr("astra.records.write_record", refuse) - result = _make(root, "first") + result = _make(root, "first#baseline") assert result.status == "failed" assert "could not be recorded" in result.reason +def test_whatever_goes_wrong_is_returned_rather_than_raised( + root: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """The contract holds for failures nobody anticipated, not only for + the ones with a message of their own.""" + def explode(*args: object, **kwargs: object) -> None: + raise RuntimeError("nobody guarded this") + + monkeypatch.setattr(worker, "execute", explode) + + result = _make(root, "first#baseline") + + assert result.status == "failed" + assert "RuntimeError: nobody guarded this" in result.reason + + # ---- the environment gates ------------------------------------------------- -def test_an_environment_that_moved_under_the_run_is_refused(root: Path) -> None: - """A manifest may not claim an environment that had already been edited - by the time the recipe ran.""" +def test_an_environment_that_moved_before_the_recipe_is_refused(root: Path) -> None: + """A record may not claim an environment that had already been edited + by the time the recipe ran — and the recipe is not run to find out.""" result = worker.execute( - root, _task(root, "first"), {}, _context(root, env_version="sha256:from-another-run") + root, + _task(root, "first#baseline"), + {}, + _context(root, environment="urn:lightcone:env:sha256:from-another-run"), + ) + + assert result.status == "failed" + assert "environment changed" in result.reason + assert not (root / _FIRST).exists() + assert not _recorded(root) + + +def test_an_environment_that_moved_under_the_recipe_is_refused( + root: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """The recipe ran, so there are bytes — and nothing says which + environment made them, so nothing is recorded.""" + task, context = _task(root, "first#baseline"), _context(root) + answers = iter([context.environment]) + monkeypatch.setattr( + identity, "environment", lambda _: next(answers, "urn:lightcone:env:sha256:edited") ) + result = worker.execute(root, task, {}, context) + assert result.status == "failed" assert "environment changed" in result.reason - assert assets.read(root / "results/baseline/.first.manifest.json") is None + assert (root / _FIRST).read_text() == "one\n" + assert not _recorded(root) # ---- what a recipe may touch ----------------------------------------------- +_TAMPERS = _SPEC.replace("echo {decisions.word}", "echo tampered > src/injected.py && echo one") + def test_a_recipe_cannot_write_outside_the_results_tree(root: Path) -> None: """The tree is read-only apart from the directory the output lands in. Sibling outputs inside that directory are *not* carved out — an output is one file, and a mechanism can only grant creating it through the - directory that will hold it; the manifest's content hash is what says + directory that will hold it; the record's content hash is what says whether an output's bytes are its own.""" - from lightcone.engine import sandbox - if sandbox.detect().capability.kind == "none": pytest.skip("no sandbox mechanism on this host") - (root / "astra.yaml").write_text( - _SPEC.replace( - "echo one > {output}", "echo tampered > src/injected.py && echo one > {output}" - ) - ) + (root / "src").mkdir() + (root / "astra.yaml").write_text(_TAMPERS) - assert _make(root, "first").status == "failed" + assert _make(root, "first#baseline").status == "failed" assert not (root / "src" / "injected.py").exists() @@ -377,53 +555,14 @@ def test_that_write_would_have_succeeded_unsandboxed( ) -> None: """The mutation check. Without it the test above would pass on a host that enforces nothing, and pin nothing at all.""" - from lightcone.engine import sandbox - monkeypatch.setattr(sandbox, "detect", Unavailable) - (root / "src").mkdir(exist_ok=True) - (root / "astra.yaml").write_text( - _SPEC.replace( - "echo one > {output}", "echo tampered > src/injected.py && echo one > {output}" - ) - ) + (root / "src").mkdir() + (root / "astra.yaml").write_text(_TAMPERS) - assert _make(root, "first").status == "ok" + assert _make(root, "first#baseline").status == "ok" assert (root / "src" / "injected.py").read_text() == "tampered\n" -def test_a_recipe_can_read_an_annexed_input(analysis: Callable[..., Path]) -> None: - """Declared inputs live in the annex. With `filter=annex` the working - tree holds the real bytes, so the project read root covers them — but - that is worth pinning, since it is what every recipe with an input - depends on.""" - from lightcone.engine import dataset - - spec = """ - version: "0.0.13" - name: analysis - - inputs: - - id: catalog - type: data - source: data/catalog.txt - - outputs: - - id: fit - type: metric - format: txt - inputs: [catalog] - recipe: - command: cat {inputs.catalog} > {output} - """ - root = analysis(spec, files={"data/catalog.txt": "measured\n"}) - dataset.save(root, [root / "data"], "the catalog") - - result = _make(root, "fit") - - assert result.status == "ok", result.reason - assert (root / "results/baseline/fit.txt").read_text() == "measured\n" - - # ---- the entry point the run record names ---------------------------------- @@ -443,17 +582,16 @@ def test_the_worker_module_imports_neither_click_nor_rich() -> None: def test_the_module_runs_one_task_and_commits_nothing(root: Path) -> None: """What `datalad rerun` invokes. It leaves the tree dirty by design — which is exactly why it is not an `lc` verb.""" - from lightcone.engine import dataset - proc = subprocess.run( - [sys.executable, "-m", "lightcone.engine.worker", "baseline/first"], + [sys.executable, "-m", "lightcone.engine.worker", "first#baseline"], cwd=root, capture_output=True, text=True, ) assert proc.returncode == 0, proc.stderr - assert (root / "results/baseline/first.txt").read_text() == "one\n" + assert (root / _FIRST).read_text() == "one\n" + assert _record(root, _FIRST_RECORD)["address"] == "first#baseline" assert dataset.status(root) @@ -463,11 +601,11 @@ def test_the_module_reruns_unconditionally( """A rerun is a rerun: the caller has already said what they want, and a staleness check would answer a question nobody asked.""" monkeypatch.chdir(root) - _make(root, "first") - (root / "results/baseline/first.txt").unlink() + _make(root, "first#baseline") + (root / _FIRST).write_text("untouched\n") - assert worker.main(["baseline/first"]) == 0 - assert (root / "results/baseline/first.txt").exists() + assert worker.main(["first#baseline"]) == 0 + assert (root / _FIRST).read_text() == "one\n" def test_the_module_converges_the_environment( @@ -476,39 +614,62 @@ def test_the_module_converges_the_environment( """A rerun checks out the lock but never `.venv`, and the recipe's own `uv run --no-sync` silently creates an *empty* environment where none exists — so the entry point syncs before anything executes, or a fresh - clone's rerun would record the lock's `env_version` over a recipe that + clone's rerun would record the lock's environment over a recipe that ran in a bare venv.""" - import shutil - monkeypatch.chdir(root) shutil.rmtree(root / ".venv") - assert worker.main(["baseline/first"]) == 0 + assert worker.main(["first#baseline"]) == 0 assert (root / ".venv").exists() - assert (root / "results/baseline/first.txt").read_text() == "one\n" + assert (root / _FIRST).read_text() == "one\n" -def test_the_module_refuses_an_argument_it_cannot_use(root: Path) -> None: +def test_the_module_reads_what_its_task_read_from_the_records_in_the_tree( + root: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """There is no graph in flight to take an upstream's result from, so + its record stands in — and not a rehash of its file, for the reason + no dependent ever rehashes.""" + monkeypatch.chdir(root) + first = _make(root, "first#baseline") + (root / _FIRST).write_text("not the bytes the record gives\n") + + assert worker.main(["second#baseline"]) == 0 + assert _record(root, _SECOND_RECORD)["generated_by"]["inputs"] == _reads_first(first) + + +def test_the_module_refuses_a_task_whose_upstream_has_no_record( + root: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + monkeypatch.chdir(root) + + assert worker.main(["second#baseline"]) == 2 + assert "no valid record" in capsys.readouterr().err + assert not (root / _SECOND).exists() + + +def test_the_module_refuses_anything_but_one_task() -> None: assert worker.main([]) == 2 - assert worker.main(["first"]) == 2 + assert worker.main(["first#baseline", "second#baseline"]) == 2 def test_the_module_refuses_an_output_that_does_not_exist( root: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] ) -> None: + """A result's name alone is not a task: a rerun makes exactly one.""" monkeypatch.chdir(root) - assert worker.main(["baseline/nothing"]) == 2 - assert "no output `baseline/nothing`" in capsys.readouterr().err + + assert worker.main(["nothing#baseline"]) == 2 + assert "no output `nothing#baseline`" in capsys.readouterr().err + assert worker.main(["first"]) == 2 def test_only_the_lookup_can_blame_the_target( root: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] ) -> None: - """The whole body used to sit under one `except KeyError`, so anything - raising one inside astra's validation or resolution was reported as - "no output ". This is the entry point every `[DATALAD RUNCMD]` - record names, so a rerun failing for an unrelated reason misdiagnosed - itself.""" + """This is the entry point every `[DATALAD RUNCMD]` record names, so a + rerun failing for an unrelated reason must not misdiagnose itself as + "no output ".""" monkeypatch.chdir(root) def explode(_: Path) -> None: @@ -516,23 +677,20 @@ def explode(_: Path) -> None: monkeypatch.setattr(worker.plan, "build", explode) with pytest.raises(KeyError): - worker.main(["baseline/first"]) + worker.main(["first#baseline"]) assert "no output" not in capsys.readouterr().err -def test_a_declared_input_that_is_not_there_names_itself( +def test_the_module_refuses_an_analysis_astra_rejects( analysis: Callable[..., Path], monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str], ) -> None: - """`data_version` reports an absent path with the OS's own exception, - which would unwind as a traceback at whoever is reading a rerun.""" - spec = _SPEC.replace( - " recipe:\n command: echo one > {output}", - " inputs: [catalog]\n recipe:\n command: echo one > {output}", - 1, - ) - root = analysis(spec) + """A declared input that is not there is ASTRA's to report, and it + reaches whoever is reading a rerun as a refusal, not a traceback.""" + root = analysis(_READS, files={"data/catalog.txt": "measured\n"}) monkeypatch.chdir(root) + (root / "data/catalog.txt").unlink() - assert worker.main(["baseline/first"]) == 2 - assert "the declared input `catalog` cannot be read" in capsys.readouterr().err + assert worker.main(["fit"]) == 2 + assert "does not validate" in capsys.readouterr().err + assert not (root / "results/fit.txt").exists()