[aboucaud]

What

This PR is stacked on #255 and pairs with LightconeResearch/lightcone-evals#3. The report job no longer needs to know what a trajectory looks like or what to ask about it; the task now owns both.

• The rows come from what the task's verifier recorded in result.json: reward, agent_steps and skills_loaded. The jq walk over the ATIF trajectory is gone. require_skills still gates, now on skills_loaded.
• The pain points come from harbor analyze, which judges each trial by the task's rubric.toml, or by Harbor's default rubric if a task has none. That replaces the inline pain-point prompt. The PR comment shows each criterion as ✅ / ❌ / ➖ with its explanation, under the analyzer's summary.
• What changed vs main is the one prompt left in the workflow, option (b). It runs claude -p over this run's trajectory, main's trajectory, the rubric analysis and the change under test. It only runs when a main run of that leg is on record. It stays in CI because a single trial can't see another run.
• The report job now installs Harbor. Its timeout goes from 20 to 45 minutes, because harbor analyze runs an analyzer container per leg.
• CLAUDE.md describes the split.

Size

eval.yml stays at about the same length. What changes is what it needs to know:

• Removed: the trajectory-schema logic (tool names, the SKILL.md matching) and the pain-point prompt.
• Added: a Harbor install step, and the smaller comparison prompt that option (b) keeps.

Order

1. Merge snae-build: record agent steps and skills in the verifier, add a rubric for harbor analyze lightcone-evals#3 first. Until then, tasks_ref: main serves the old task:
   • The rows carry no agent_steps, so the step gate has nothing to compare. It isn't active yet anyway: it needs three main runs on record.
   • skills_loaded is unknown, which counts as "not loaded" under require_skills. That would fail agent-skills#56. Nothing fails here, because require_skills is off by default.
   • harbor analyze falls back to its default rubric.
2. Then this PR.

Verified

• actionlint is clean.
• I took the summary-and-gate and render steps out of the workflow and ran them on a fixture carrying the new reward keys:
  • steps and skills come through;
  • skills_loaded: 0 adds "no skill was loaded" under require_skills;
  • a sample analysis.json renders as a summary plus ✅ / ❌ / ➖ checks.
• Not run here: harbor analyze itself, because this sandbox's proxy blocks the image pull. The eval run on this PR is its first real test.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Y9FLtiiMxVhhkvZjd9SQp4

[aboucaud]

This is superseded by LightconeResearch/lightcone-evals#5. The eval workflow now lives in lightcone-evals, beside the task it runs. This PR's changes are folded in there, along with Francois's simplifications:

• --skill from a git source;
• the oracle in the agent job;
• reading reward.json directly.

I'm closing this PR. #255 becomes the caller.

---

Generated by Claude Code