Your code changes fast. Your docs fall behind. spec-eval keeps them honest.
The trust layer for spec-driven development — iterate fast, keep the spec and code in agreement.
A trust-eval for your specs — coverage, drift, and sufficiency, measured against the code.
Get started · Tutorial · No API key? · Second opinion · FAQ
A spec is short for specification — a plain-English description of what your code should do. spec-eval keeps one beside your code — per file, or per folder — and checks the two still agree, so the spec stays worth trusting. How far you lean on it is up to you. No specs yet? spec-eval writes them for you.
- The problem
- What it checks
- What it creates
- Try it
- How the scores are made
- Reading the scores
- Higher stakes? Get a second opinion
- Proof it works
- More
- FAQ
- Feedback
- Acknowledgements
Specs go stale the moment code moves on — and one stale spec is enough to stop a reader trusting all of them. AI coding agents make code move faster than any hand-maintained doc can follow.
spec-eval treats specs like code: run a check — via a chat prompt, an ordinary command, or a CI step — and the spec is measured against the source. A spec you can check is a spec you can trust.
Three simple questions:
| Check | The question it answers | How |
|---|---|---|
| coverage | Does this code have a spec at all? | Just counts files. Free — no AI, no key. |
| drift | Does the spec say something the code doesn't do? | An AI reads each spec and its code and lists the clashes. |
| sufficiency | How completely does the spec capture the code's behavior? | An AI lists what the spec misses and scores it 0–1 — an indicator, not a guarantee. |
Your code is the source of truth — the spec is what gets graded against it.
Everything it writes is plain markdown, right next to your code:
your-project/
├─ src/
│ ├─ parser.py
│ ├─ parser.md ← generate: a spec beside each code file
│ ├─ api.py
│ └─ api.md
├─ OVERVIEW.md ← optional overview of the spec set (--overview repo)
└─ spec-reports/ ← the checks write here
├─ coverage.md which files have a spec
├─ system-context.md which external systems the code talks to (with file:line evidence)
├─ report.md drift findings
├─ sufficiency.md 0–1 completeness per spec
├─ runs.jsonl run history
└─ SPEC-HEALTH.md one-page scorecard (ask the skill to write it, or edit by hand)
Prefer a guided walkthrough? Getting-started tutorial →
The commands — in the order you'll typically use them:
- coverage — which files have no spec yet? (free — no AI)
- context — which external systems does the code talk to (S3, databases, partner APIs)? (free — no AI)
Strong support for the mainstream stacks (Python, JS/TS, Java, Kotlin, C#, Go, Rust, Swift, Ruby) — the
exact list lives in
spec_eval/syscontext.md. - generate — writes a plain-English spec beside each file that has none (uses AI)
- sufficiency — how completely does each spec capture the code? Scored 0–1 (uses AI)
- audit — the drift check: does any spec clash with the code? Run it at every milestone (uses AI)
None of them change your files — reports go to spec-reports/. The exception is generate, which writes new
specs beside your code: review them and commit the keepers like code.
The (uses AI) commands (generate, sufficiency, audit, and diagram) need an AI behind them —
coverage and context never do. Three ways to run them, least setup first:
Ask the coding agent you already have (Claude Code, Copilot, Cursor, …) — nothing to install, no key, answers
land in the chat. Copy a block, adjust the {{slots}} if you want, paste it — each slot shows its default
after =, and the agent echoes the settings for your OK before it writes.
The authoring skill carries the spec structure
(the INV-* invariant and AC-* acceptance-criteria tables come from its rubric and templates):
Read {{SKILL = https://github.com/benjaminjones/spec-eval/blob/main/skills/spec-authoring/SKILL.md}}
and its templates/ folder, then follow it to write specs for: {{TARGET = ./ (a file, a folder, or the whole project)}}
Settings — defaults below. Before writing anything, echo TARGET and these three settings back in one
message — each setting with its full option list from the comments, so I can pick in my reply — and
wait for my OK:
- layout: per-file # per-file (a spec beside each file: src/x.py -> src/x.md) | per-dir (one spec per folder) | per-pair (folders you map in a config)
- overview: none # none | repo | per-dir | both — none = no overview files; any other value adds them (both = top-level OVERVIEW.md + a README.md per folder). A repo-level OVERVIEW.md carries an Architecture (data flow) Mermaid diagram + a System context section (the external systems the code talks to, when any are observed); per-folder READMEs are a lean index.
- reports: chat only # chat only (results land in the chat) | save (also save the results to spec-reports/)
Do one scoped job at a time (one file, or one folder) so a small-context agent doesn't run out of tokens.
The checking skill grades what exists (coverage + drift):
Read {{SKILL = https://github.com/benjaminjones/spec-eval/blob/main/skills/spec-check/SKILL.md}}
and follow it to check specs for: {{TARGET = ./ (a file, a folder, or the whole project)}}
Before checking, echo TARGET and these two settings back in one message — each setting with its full
option list from the comments, so I can pick in my reply — and wait for my OK:
- run both passes, in order: 1) coverage (which files have no spec), 2) drift (do the specs match the code)
- double-check: no # no | yes (re-read each spec and drop findings it doesn't actually support)
- reports: chat only # chat only (results land in the chat) | save (also save the results to spec-reports/)
Check one folder at a time so a small-context agent doesn't run out of tokens.
Tip
Cloned the repo? Point {{SKILL}} at your local path/to/spec-eval/skills/…/SKILL.md. With the remote
default, your agent may ask you to approve each fetch (the skill, then its templates) — that's normal; the
local path skips the prompts. If your prompt and
the skill disagree, your prompt wins — the skill carries the method, your slots carry the choices.
The overview values: none (default) — no overview files · repo — one OVERVIEW.md at the top of the
scanned path · per-dir — a README.md overview in each folder with 2+ modules · both — repo + per-dir.
(layout places the specs; overview only adds overview files.)
Tip
Author and check in separate sessions. A checker that still remembers its authoring decisions tends to echo them instead of re-reading the code — the same reason a PR isn't reviewed by its author. Write specs in one session; open a fresh one to check them.
Make it a standing command (optional, one-time) — copy the skills/ into your agent's skills
folder (Claude Code: .claude/skills/). Then the prompt is the command:
mkdir -p .claude/skills && cp -r path/to/spec-eval/skills/* .claude/skills/Which files have no spec yet? # coverage
Write a spec for src/parser.py # generate
How completely do my specs cover my code? # sufficiency
Check my specs against my code # audit (drift)
Show me the architecture diagram # diagram
Install once, then add --model claude-code to any (uses AI) command — spec-eval routes through the
claude CLI you're already logged into (no key; your ANTHROPIC_API_KEY is hidden from the call, so it can't
accidentally bill the paid API). The same commands, in the same order as the list above:
pip install spec-eval # needs Python 3.9+; or, from a clone: pip install -e .
spec-eval coverage ./your-project # free — no AI, no key
spec-eval context ./your-project # free — no AI, no key
spec-eval generate ./your-project --model claude-code
spec-eval sufficiency ./your-project --model claude-code
spec-eval audit ./your-project --model claude-code
spec-eval diagram ./your-project --model claude-code # Mermaid architecture diagram → stdout (--write to save)Every terminal run writes its reports to spec-reports/ (plus a runs.jsonl history line).
Export a key and run any command above without --model claude-code — an API key is the default. This is
the option for CI, the checks that run automatically on every push (store the key as a repo secret):
export ANTHROPIC_API_KEY=sk-ant-... # or OPENAI_API_KEY / GOOGLE_API_KEYNote
A typical first run is coverage → generate → sufficiency, then at every milestone after: audit — plus
sufficiency when you've added behavior. The tutorial has step-by-step guides for
starting with no specs and for existing docs.
Tip
Version control is also your preview. The three checks never modify your project; generate adds
ordinary new files — review them with git diff, edit what's off, and drop any you don't want with
git checkout -- <file>. (A full generate --overwrite rewrites everything, so save it for first setup.)
A few things worth knowing:
- No config needed — by default each spec pairs with the code file of the same name beside it (
parser.md↔parser.py). Prefer one spec per folder?--layout per-dir(FAQ). - Any language, three providers — swap the model with
--model openai:…,google:…, orclaude-code(Anthropic is the default); code is read as plain text. Want another provider? Open an issue. - Your other docs are safe — by default only a
.mdnext to code with the same name counts as a spec, so your READMEs are left alone. (Folder specs and overview files only appear if you opt in with--layout/--overview.) - What's shared — to grade a spec, its code and text go to your AI provider. That's it. (
coverageandcontextsend nothing — they run fully on your machine.)
Keep it honest on every commit — add one line to a git hook (a script git runs before each commit) so new code always needs a spec; copy-paste setup in the tutorial:
spec-eval coverage . --min 90 # fails the commit if coverage drops below 90% (free — no AI)Three scores, three simple ideas:
- coverage = spec files you have ÷ spec files you need → a percent. Just counting, no AI. (9 of 10 code files have a spec → 90%.)
- drift = the spots where the AI reader found the spec and the code flat-out disagreeing. The number
counts the ones it marked high or medium — small ones are still listed, just not counted.
0means it found nothing this run. It is not proof that your spec and code agree. - sufficiency = how much of what the code does is actually written in the spec, from
0to1. (1.0= it's all there;0.1= almost none of it is.)
Those three are per spec. The repo's one sufficiency score is just their average — each spec's
0–1, added up and divided by how many specs were scored. (coverage is already one repo-wide %; drift
adds up the high and medium ones across specs — and with --verify, only the ones that survived the check.)
coverage is pure counting. The other two need an AI reader: it reads each spec next to its code and grades
the pair against a fixed rubric — one model call per pair (audit --verify adds one more, but only for pairs that had findings). Two things follow from that:
Note
AI scores wobble: the same spec might get 0.78 one run and 0.72 the next, like two teachers grading the same essay. Watch the trend, not the decimal — a jump like 0.72 → 0.85 is a real improvement.
Important
AI calls spend tokens — through your API key, your Claude subscription with --model claude-code, or
your coding agent's own usage if you asked in chat. Every CLI run prints its exact call and token counts.
Want the exact rules behind each score? spec-eval specs its own scoring — read
spec_eval/coverage.md (how coverage counts),
spec_eval/rubric.md (the drift rules), and
spec_eval/sufficiency.md (how the 0–1 score is decided).
Each run writes a short report to spec-reports/ (a .md you read, a .json for tools):
- coverage — the % of your code that has a spec, and which files don't. (live example: coverage.md)
- drift — each place the AI reader found a spec and the code disagreeing: the code line and the spec line it quoted, plus a suggested fix.
0means this run found nothing. (live example: report.md) - sufficiency — a
0–1score per spec (worst first), listing what's missing with a searchable code pointer (file.py (function)).1.0= the grader found nothing missing — guidance, not a guarantee. (live example: sufficiency.md)
Important
A finding can be wrong. The AI reader sometimes reports a clash that isn't really there. That is why every finding quotes the code line and the spec line it rests on — read the quote before you change anything. If the quote doesn't back up the finding, the finding is the mistake. Leave your spec alone.
Not sure about the findings? Double-check them. Add --verify and a second AI pass re-reads your spec and
throws out findings the spec doesn't actually support:
spec-eval audit . --verifyIt only ever removes findings, never adds any, so the list of findings that stand can get shorter but never longer. Thrown-out ones still show up — crossed out, with the reason and the spec line — so you can disagree with it. It costs one extra AI call per file that had findings, which is why it's off unless you ask.
Why a second pass and not a more careful first one? A reader that has talked itself into something stays talked into it — longer answer in the FAQ.
The three examples are spec-eval grading itself, rolled up in SPEC-HEALTH.md.
Every command also appends a line to runs.jsonl — timestamp, git commit, scores — so you can track change over
time.
drift and sufficiency are AI judgments — one model can miss something or be too generous. For a spec that needs extra attention — a critical path, complex design, or compliance code — run the
same check with two different AI vendors and compare. Give each run its own
--out folder (otherwise the second run overwrites the first report):
spec-eval audit ./your-project --model anthropic:claude-opus-4-8 --out spec-reports/claude
spec-eval audit ./your-project --model google:gemini-3.5-flash --out spec-reports/gemini
diff spec-reports/claude/report.md spec-reports/gemini/report.md # or read them side by sideHow to read two reports:
- Both flag it → almost certainly real. Fix these first.
- Only one flags it → a judgment call. Read the quoted evidence and decide.
- Sufficiency scores differ a little → normal wobble. Compare the gaps lists, not the decimals — the same missing behavior named by both graders is the signal.
"Normal wobble" is easy to say and easy to get wrong in both directions. A single sufficiency score is one draw, so before reading a gap between two vendors, find out how much the same vendor disagrees with itself:
spec-eval sufficiency . --reps 3 --model claude-code --out spec-reports/claude
spec-eval compare spec-reports/claude/sufficiency-reps.json \
spec-reports/claude/sufficiency-reps.jsonComparing a run against itself gives a difference of zero by construction — that is not the point. The
point is noise share, which tells you how much of the score is the grader talking to itself. Do this
first: if the noise is large, a between-vendor gap has nothing to sit on, and you have learned that without
a second API key.
Then, with two vendors:
spec-eval sufficiency . --reps 3 --model claude-code --out spec-reports/claude
spec-eval sufficiency . --reps 3 --model google:gemini-3.5-flash --out spec-reports/gemini
spec-eval compare spec-reports/claude/sufficiency-reps.json \
spec-reports/gemini/sufficiency-reps.jsoncompare runs after scoring, on its own. It is not part of a sufficiency run and it does not read
audit output — it reads the retained sufficiency reps and does arithmetic. It makes no model calls, so
it is free, and you can re-run it with a different margin as often as you like without re-scoring anything.
What it prints, and how to read it:
| Smallest detectable effect | printed first, always. If it is bigger than the difference you care about, this run cannot settle the question — no matter what the rest of the output says |
| delta ± SE | the vendor difference against its own standard error. A delta smaller than its SE is not a difference |
| equivalent within ±X | true only when the whole interval fits inside a margin you set before running. "No significant difference" is not the same claim, and this tool will not make it for you |
| noise share | rises with noise. 0.35 means about a third of the variation is the grader disagreeing with itself |
| per-pair deltas | exploratory. Across a dozen pairs the largest one is expected to look big by chance |
--reps defaults to 1, which writes the usual sufficiency.json; compare accepts that too and reads it
as a single run — you just get no noise share, because one observation cannot show spread.
Each vendor reads its own key from the environment — ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_API_KEY —
so put both keys in one .env and each run picks up the one it needs. (Want one fewer key? Swap either side for
--model claude-code, which uses your Claude Code login instead.)
- spec-eval specs itself — every module in
spec_eval/that needs a spec has one (11 of 11; two plumbing files are left out, and the receipt says which): coverage 100% · drift 8 · sufficiency ≈0.86 (receipt — the exact, dated score, and which model made it). Coverage is plain counting, so 100% is exact. Drift is what one AI reader found on one day — 17 raised, 7 thrown out by a second pass, 8 serious ones left standing. The drift report beside the receipt lists every one of them, so you can check it yourself.
- Works with spec-driven development. SDD's goal is a spec you can trust, and spec-eval is that trust layer. Use it three ways: grade the specs you already have, generate specs for code that has none, or check as you write (with the skills). Spec-first, code-first, or in between — it meets you where you are.
- Grow the spec and the code together. The best version of "in between": draft a little spec, write a little code, check they still agree — repeat. The two can even move in parallel (you shape the spec while your coding agent writes the code; the check is where they meet), so the spec is born accurate instead of written up afterward.
- Just want to vibe code? Fine — build first, check later, automatically: a pre-commit gate and a GitHub Action can run the checks for you — see Make it routine.
- Higher-stakes spec? Run it past two vendors and compare — see Higher stakes? Get a second opinion above.
Common questions — from "which command do I run first?" through costs, layouts, and score accuracy — live in FAQ.md.
Found a bug, a bad score, or a spec that misses the point? Open an issue — a report snippet and the command you ran is plenty.
Built on the Anthropic, OpenAI, and Google SDKs (MIT / Apache-2.0) + PyYAML.
Terms you may be searching for: software specification · documentation drift · living documentation · docs as code · backfill documentation · spec-driven development (SDD) · agentic development · AI-assisted coding · vibe coding · vibe engineering · AI code documentation · LLM-as-judge / trust eval. MIT licensed.