Repository navigation
docs: reconcile internal contradictions across the documentation - #59
Merged
Merged
Conversation
The Definitions table enumerated authored, skipped and failed. generate_repo also emits stray, for markdown that appeared on disk without being a declared target (authoring.py:650), which INV-15 and AC-18 both describe. A reader taking the row as the closed set would not have handled it. The row now names all four, and says which one belongs to a record that is not a declared target. skipped also gains its second cause: a per-dir overview below overview_min_files is recorded skipped with a note (authoring.py:625), not only a file that was already there. The generate result contract said one record per target. It is one per target plus one per stray write, and AC-18's example record now shows the code key the implementation always sets.
The contract line omitted class and marked code_ref and doc_ref optional with
a trailing ?. Both parse paths emit class (audit.py:85, :94), constrained to
drift or stale, and both always write code_ref and doc_ref - the keys are
present and the values may be null, which is what INV-5 already promised. The
? notation said the opposite.
The contract now reads class in {drift,stale} and code_ref|null, doc_ref|null,
matching the repo's existing notation for a present-but-nullable field. INV-5
and AC-4 carry the same seven keys.
One acceptance criterion had been added under an ID the table already used;
it takes the next free number.
…finition that matches the stale rule Two acceptance rows had been added under IDs the table already bound to other criteria. Each takes the next free number in addition order, so every ID names one criterion. The drift-load definition counted high and medium findings not withdrawn by the verification pass. It also excludes stale findings - the Stale row says so, and report.py:18-20 implements it - so the definition now names both exclusions.
…-severity breach The acceptance row graded a doc-versus-code default as high severity. The rubric (rubric.py:19-21) reserves high for a measurable guarantee the code breaks, and rubric.py:28-31 says a declarative value cannot be violated, only outdated - it is reported as stale, not counted as drift, and its severity measures how misleading the value is rather than how urgent a breach is. The row now says that.
Every reported gap was said to carry a severity of major or minor. The parse-failure fallback emits a gap carrying the ? sentinel instead of a grade, so both the prose and the field table now name it as the one exception.
The documentation-domain skip said api.github.com is kept while also stating the match is at a domain boundary, which would skip it; the code keeps it through an explicit host exception the text never mentioned. The rule now names the exception. tables_digest was described as hashing all detection tables. It covers the narrower set INV-14 and the fingerprint section already name.
Every command was said to create --out and write both a JSON and a Markdown artifact; the command table shows commands that write one and commands that write none. The exit-code sentence named a single condition where the paragraph below it names two. Both now match the table.
compare makes no model calls. The inputs it consumes are the output of runs that did. The sentence now separates the two so it does not read as a claim about the data.
The prose said token counts come back; the matrix records a provider that does not return them. The prose now matches the matrix.
The block described one setting while specifying two, and the option lists in the prompt disagreed with the comment above them. The AI-command enumeration did not match the per-command annotations. All three now match cli.py.
…matches the CLI The paragraph promised two habits and listed three. It also repeated the claim that every command creates --out, which the command table contradicts.
The answer named a different set of key-free checks than the command table does. coverage and context need no key; the rest do.
The walkthrough described exit behaviour that the CLI's own exit-code table contradicts.
The skill promised the same files the CLI produces and then listed one the CLI does not write - SPEC-HEALTH.md is authored by hand, as the sync note beside it already said.
…claim is scoped A stated total disagreed with the enumeration that followed it, and a claim about the only thing that stamps was written unscoped where two different stamps exist.
Layer 1, per TESTING.md - contracts framed as invariant checks. The finding key-set test asserted only that both parse paths AGREE. Both could grow a key together and still agree, so it never pinned what the set is. It now asserts the seven keys audit.md INV-5 enumerates, which is what lets a consumer index any of them unconditionally. Its docstring said six. The status vocabulary had no executable form at all, and drifted: stray was emitted by generate_repo and missing from the Definitions row. No test asserted stray as a status - authored, skipped and failed appeared across the suite and stray appeared nowhere. The new contract test reads both files and fails in either direction: a value emitted but undocumented, or documented but no longer emitted. It was checked against the pre-fix document and fails there, so it is not inert, and it carries a third assertion guarding against both set comparisons passing vacuously if the extraction silently matched nothing.
benjaminjones
added a commit
that referenced
this pull request
Sep 30, 2026
All four measuring commands re-run against current main -- coverage, context, audit --verify and sufficiency -- and SPEC-HEALTH's verdict, fingerprint and run history updated from them. Every number comes from one commit this time, unlike the previous row which spanned three. coverage 100%, 12/12 modules (was 11/11) drift 12 high/medium, 7 withdrawn (was 8 and 7) sufficiency 0.82 average across 12 pairs (was 0.86 across 11) The audit was run WITH --verify. Without it the raw count is 19, which is not comparable to the previous row's verified 8, and a number needing a paragraph to explain why it cannot be compared is worth less than the extra calls. Drift rose 8 to 12 on a like-for-like basis, and the tree it measures is not the tree the last row measured. Since 2026-08-06 the package gained the compare subcommand and module, a process-wide call ceiling in providers, and --max-calls and --reps on sufficiency -- none of them specced. That is where the rise sits: cli 0 to 3 with sufficiency falling 0.86 to 0.62, compare arriving at 0 to 2, providers 0 to 1. The code grew and the specs did not follow, which is the finding rather than a regression in the checker. ONE OF THE TWELVE WAS INTRODUCED BY THE DOCUMENTATION PASS IN #59 AND IS LEFT IN THE COUNT. cli.md's Artifacts and logging paragraph read "Every command creates --out..." and was amended to "Every command except diagram..." to settle an internal disagreement with the command table. The code registers seven subcommands, and compare writes only compare.json with no Markdown and no runlog.append_run, so the sentence became more precise and stayed wrong about a second exception the document never listed. It grades high because a more specific false claim is worse than a vague one. It is recorded rather than quietly repaired, on the same principle the previous row states: re-running to lower the number would score the gap this run named. The repair belongs in its own change. The 7 withdrawals match the previous run's count; each cites a ground from the closed set and a document line that exists. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
benjaminjones
added a commit
that referenced
this pull request
Oct 9, 2026
A semicolon separates statements in Mermaid, and sequence message text is unquoted — so `A->>B: one thing; another` parses as two statements and the block fails to render, with the second half's first word read as an actor. Scoped to sequence blocks. A flowchart label is quoted, so a `;` there is safe; rewriting it would change prose to fix a problem that grammar does not have. An em dash rather than `;`: the entity does make the block parse, but it renders literally as `&;`, which is readable-but-wrong and worse than a clear substitution. A semicolon in this prose always joins two clauses. 7 tests, including that a clean block is returned byte-identical, that a colon inside the text is not treated as the boundary, and that a participant declaration is not mistaken for a message. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Each document contained a statement that disagreed with another statement in the same document. This reconciles them.
No behaviour changes. No production code changes. One commit per document.
What changed
spec_eval/authoring.mdstatusrow omittedstray;skippedlisted one cause of two; result contract said one record per targetspec_eval/audit.mdFindingcontract omittedclassand marked always-present keys optional; one acceptance ID bound twicespec_eval/report.mdspec_eval/rubric.mdspec_eval/sufficiency.md?sentinel on the parse-failure gap was unmentionedspec_eval/syscontext.mdtables_digestdescribed too broadlyspec_eval/cli.mdspec_eval/compare.mdspec_eval/providers.mdREADME.md,CONTRIBUTING.md,FAQ.md,GETTING-STARTED.md--outand API-key claims disagreeing with the command tableskills/*/SKILL.mdTests
Layer 1 per
TESTING.md— contracts framed as invariant checks.test_unparsed_fallback_carries_the_same_key_setasserted only that the two parse paths agree; both could grow a key together and still pass. It now pins the seven keysINV-5enumerates. Its docstring said six.test_status_vocabulary_sync.py(new) — thestatusvocabulary had no executable form and drifted. It reads both files and fails in either direction: a value emitted but undocumented, or documented but no longer emitted. Verified against the pre-fix document, so it is not inert.Checks
ruff check .passes ·pytest336 passed (333 before) · no duplicate acceptance IDs remain inreport.mdoraudit.md·spec-reports/andtests/fixtures/untouched, since they record what a document said during a past run.🤖 Generated with Claude Code