Skip to content

Updating lightcone-cli to new version of ASTRA - #268

Draft
EiffL wants to merge 3 commits into
mainfrom
astra-update
Draft

EiffL wants to merge 3 commits into
mainfrom
astra-update

Conversation

@EiffL

@EiffL EiffL commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Why

ASTRA 0.1 and the rebuilt astra-tools change what a spec is and who answers what. Universes live in astra.yaml; the spec fixes where a result and its provenance record go; and astra-tools now provides the plan, the staleness checker and record-writing building blocks that lc used to implement itself. This PR makes lc the runner on top of astra.api and nothing more.

What

Engine

  • Planning (plan.py): asks astra.api.plan instead of resolving the spec. A task is one planned result, keyed by its address: fit#baseline, fit for an output built once, stage.fit#u1 for a driven declared analysis, stage/fit#v1 for a result that belongs to one. Paths are measured from the project root, which may sit below the repository's top. A recipe runs in its own analysis's folder.
  • Staleness (assets.py): asks astra.api.check instead of hashing recipes and inputs (definition_version is gone). lc adds only behind (the record names an earlier environment) and the git-history foreign-write test. States stay current / behind / stale.
  • Records (worker.py): each result gets an ASTRA record under .astra/records/, naming the environment urn:lightcone:env:<env_version>. An astra.environments entry point tells astra check that IRI is recreatable.
  • No manifest sidecar. The record is the only file lc writes beside a result. The rendered recipe, recorded decisions, git remote, uv version, sandbox hermeticity attestation and image id are no longer kept.
  • Cascade stop: a byte-identical rebuild still stops the cascade. A task downstream of a possible remake re-asks the checker in the worker once its upstreams have landed.
  • Refusals: an analysis declared with a commit, and a result owned by a nested repository, are refused with a clear error rather than built.

Around it

  • lc init scaffolds the minimal 0.1 spec; .gitattributes also annexes the results/ of an analysis in a folder below the project.
  • RO-Crate: runs are timed and attributed from records; each run action carries the task's command and decisions as the spec now stands; no ContainerImage entity.
  • lc run discovers declared inputs through astra.api.bind, across declared analyses.
  • lc view server routes read records; the paper route serves PDFs from astra's new cache. /lightcone/api/run no longer returns recipe.
  • CLAUDE.md is rewritten as a lean guide (the first commit) and updated to this model.

Dependencies

ASTRA 0.1 is not on PyPI. pyproject.toml pins astra-tools (ecbe950) and astra-spec (7de4f21) by git, with an override-dependencies entry for astra-spec.

Tests

Every suite is re-expressed as 0.1 specs. Tests for what ASTRA now owns or forbids (definition hashes, directory inputs, out-of-tree inputs, universes/*.yaml) are deleted. New tests/test_composition.py runs the specification's own shapes end to end: pins, an imported universe, derived_from, when, a driven analysis, and the spec's "edit → what goes stale" table.

Locally: ruff and mypy clean; 1205 passed, 1 skipped (the lc view browser test, no Chromium).

Known gaps in this PR

  • lc view's page needs the 0.1 @astra-spec/sdk / @astra-spec/ui; the browser test will fail in CI until then.
  • .github/workflows/eval.yml and evals/tasks/snae/ are still 0.0.14 (astra-tools== grep, astra universe generate).
  • No migration for existing 0.0.14 projects (by the no-back-compat rule).
  • recipe.container / analysis container are ignored; containerized mode is still derived from [tool.lightcone.image].

Pain points against the current astra-tools

Each is something lc works around today; fixing it upstream would let the workaround go.

  • check is all-or-nothing. Every call re-validates, re-plans and computes every result's status; universes= / outputs= only filter the report, and each distinct record commit is a fresh bind. lc asks once in the driver and again only in tasks downstream of a possible remake; a wide cascade still pays one full check per task.
  • No "what needs building" / single-result question. Nothing answers "is this result unchanged given these upstream digests"; the logic is private (_Checker._changes). This is why workers now call check, and so need git and the repository.
  • plan.Result is not picklable (decisions is a mappingproxy), so it cannot cross to a Dask worker. lc copies each result into its own Task and builds Records by hand instead of using records.new_record.
  • new_record rehashes upstream results. lc must record the digest the upstream task reported, because on a bytes-free clone the file is an annex pointer. Another reason new_record goes unused.
  • Paths are from the repository's top, while a runner works from the analysis folder. lc re-bases every path (plan.relative).
  • No fetch path for results from a commit. A Read at a commit holds no repository and there is no fetch(read); lc refuses analyses declared with a commit.
  • A record edited together with its bytes passes check. lc keeps its own git-history foreign-write test.
  • Result bytes are not read through stand-ins. check uses stand-ins for inputs only, so an unfetched or locked annexed result reads absent; lc ignores bytes_status.
  • The record schema is closed, with no slot for runner attestation. lc dropped what it used to record (sandbox enforcement, image, uv version) rather than keep a sidecar.
  • api.bind cannot run offline, so a read-only probe may fetch remotes.
  • Stable surface is too small for a runner. lc imports astra.plan.render_command, astra.records, astra.hashing, astra.contract, astra.load, astra.sources.* and astra.cache, none of which astra.api promises.
  • Packaging. astra-tools names astra-spec by a sibling path, so a dependent needs a git pin plus an override; uv tool install ignores the override, and a dev engine's recorded rerun command cannot resolve.
  • The paper cache holds no metadata and cannot be listed, so lc view lost titles/authors and "any cached version" lookup.
  • api.init also creates a git repository, so lc writes api.MINIMAL itself.
  • same_computation is only a note. Identical results in several universes are built once per universe; nothing in the plan lets a runner share them safely.

🤖 Generated with Claude Code

The old file was a ~1,500-line per-layer record written while the
codebase was being re-added layer by layer. That rebuild is done, so the
file now describes the code as it stands: commands, the review rules,
and the architecture that spans modules.

Also drop the remaining rebuild wording from the engine docstring and
two test comments.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

❌ Eval

Metric Value
Outputs check skipped
Agent run skipped
Turns ?
Tool calls ?
Cost $?
Agent wall time ?
Model ?
lc status
unavailable

Full trace: agent-trace artifact on this run.

EiffL and others added 2 commits November 11, 2026 15:20
ASTRA 0.1 moves universes into astra.yaml, defines where a result and
its provenance record go, and says when a record still describes the
analysis. astra-tools now answers all of that through `astra.api`, so
lc stops deriving it and becomes the runner only.

- plan.py asks `astra.api.plan` and carries each planned result across
  as a picklable Task keyed by its address (`fit#baseline`, `fit` for an
  output built once, `stage.fit#u1` driven, `stage/fit#v1` owned by a
  declared analysis). Paths are measured from the project root. Results
  of an analysis declared with a `commit` are refused.
- assets.classify asks `astra.api.check` instead of hashing recipes and
  inputs; lc adds only `behind` (the record names an earlier
  environment) and the git-history foreign-write test.
- The worker writes ASTRA records under `.astra/records/`, naming the
  environment `urn:lightcone:env:<env_version>`, with an
  `astra.environments` entry point for `astra check`. A recipe runs in
  its own analysis's folder. The manifest keeps only what the closed
  record cannot hold.
- A task downstream of a possible remake re-asks the checker once its
  upstreams have landed, so a byte-identical rebuild still stops the
  cascade.
- Scaffold, .gitattributes (nested `results/` are annexed), the RO-Crate
  renderer, `lc run` input discovery, the `lc view` server routes and
  the docs follow.

ASTRA 0.1 is not on PyPI yet: astra-tools and astra-spec are pinned to
git commits. The `lc view` page still needs the 0.1 SDK, and the eval
workflow is not migrated.

Tests are re-expressed as 0.1 specs; test_composition.py runs the
specification's own shapes (pins, imports, derived_from, when, driving)
end to end.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Francois Lanusse <fr.eiffel@gmail.com>
… result

The manifest held what ASTRA's closed provenance record cannot: the
rendered recipe, the decisions, the git remote, the uv version, the
sandbox's hermeticity attestation and the image. None of it is kept
any more, so `.<name>.manifest.json`, `assets.Manifest` and the helpers
that existed only to fill it (`project.uv_version`,
`Runtime.manifest_image`, `Runtime.arch`) are gone, with the
`.gitattributes` lines that kept manifests out of the annex.

- A run commits two files per place, the bytes and the record; the
  foreign-write check looks at those two.
- The RO-Crate takes the repository's remote as an argument, describes
  each run with the task's command and decisions as the spec now stands,
  and no longer has a ContainerImage entity.
- The warning about an output the spec dropped is now raised from its
  orphaned record.
- `/lightcone/api/run` no longer answers `recipe`.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Francois Lanusse <fr.eiffel@gmail.com>
@EiffL EiffL changed the title Replace CLAUDE.md with a lean guide now that the rebuild is complete Updating lightcone-cli to new version of ASTRA Nov 11, 2026
@EiffL
EiffL marked this pull request as draft November 11, 2026 15:21

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant