Repository navigation
Conversation
Move the existing content under the new section layout without changing its wording, and add placeholder comments for the content that is still missing. Also fix the link to the mined invariants section, whose anchor did not match the generated heading id. Towards model-checking#4979
|
@acearyanarun @wodex1nhaoIeng the skeleton is up: the current page moved into Felipe's structure, no wording changes yet. Could you take a look and say whether the layout works for you? If it does, there are three parts to fill in so we don't edit the same lines:
|
Describe what a successful automatic harness does not cover: single monomorphization, bounded arguments, and the opt-in constructor and mined-invariant modes. Add an "Other limitations" subsection for the under-approximations that apply by default without a marker in the summary table, including the closure side effects reported in model-checking#4994. Towards model-checking#4979
Fill in the "Selecting functions" section of the autoharness reference page (towards model-checking#4979): include/exclude pattern matching against instantiated names, --list/--format, the full list of skip reasons, and generic instantiation, including Fn-bound parameters and the SIMD and const-generic restrictions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UFzbxD33o5GHieAP36Da8B
Describe the default harness timeout and unwinding bound, when Kani reminds the user of them, and how to read the selected/skipped tables, the summary table, and its "(bounded)" and "(ctor)" markers. Towards model-checking#4979
Describe the --std option, its path requirement, and that the alloc-dependent argument models are unavailable when verifying the standard library. Also note that concrete playback is not supported. Towards model-checking#4979
Show how the driver and the compiler split the work, from option handling through function selection, harness generation and code generation to running CBMC and printing the summary. Towards model-checking#4979
Update the overview and usage text that still said every argument must implement Arbitrary, align the variadic skip reason with the code, spell "autoharness" consistently, and link the Fn-bound closure models to the closure soundness caveat (model-checking#4994). Towards model-checking#4979
Fill in the bound options, contract preconditions, layout niches, UB and panic checks, contract postconditions and --check-invariants, and smooth out the mined-invariants text split by the restructure. Towards model-checking#4979
Reword the --std model availability note, and make the nested-slice and constructor-generics limits more precise. Towards model-checking#4979
Under --constructor-args, values of a type with its own Arbitrary implementation are still filtered by mined invariants, but the harness is not marked "(ctor)" (model-checking#4995). Say so in the constructor and mined invariant sections, the summary table, and the soundness caveats. Towards model-checking#4979
|
did you review these changes? do they seem helpful? @acearyanarun @Tianshu-Huang @wodex1nhaoIeng @srivatsansamraj |
There was a problem hiding this comment.
🟢 Approval recommended
Only a minor, non-blocking section-link correction remains in this documentation-only change.
1 open issue
What changed in this PR
Updates the autoharness reference to reflect current implementation and organize guidance around selecting functions, generating inputs, and interpreting verification results.
Changes:
- Documents supported argument models, generic instantiation, skip reasons, and CLI options.
- Explains verification defaults, result markers, and soundness limitations.
- Adds a contributor-oriented overview of the driver/compiler pipeline.
| File | Description |
|---|---|
docs/src/reference/experimental/autoharness.md |
Reorganizes and expands the autoharness reference, including modeling limits and architecture. |
Review effort: Balanced
Tianshu-Huang
left a comment
There was a problem hiding this comment.
I checked the page against main and everything matches, and all the #4979 items are in. The anchors are fine too, Yanjun is right about the en dash. Approving.
|
Looks good to me, but since I contributed some commits to this PR, I’m not sure whether it would be appropriate for me to review it. |
|
Hi Felipe, yes, I reviewed the updated documentation, particularly the function-selection and skip-reason sections I contributed to. I noticed two minor improvements: some CLI flags in headings render as en dashes, and the function-selection order could be clarified to include the comptime and no-body checks. Neither appears blocking. Overall, the changes look good to me! |
feliperodri
left a comment
There was a problem hiding this comment.
Thanks for the restructure. The headings follow the structure proposed in #4979, and the claims I spot-checked against the code are accurate (generic instantiation candidates and limits, skip reasons, default timeout/unwind, --list and --std behavior, unbounded slice/Vec models, mined-invariant limits, Fn model shapes).
Requesting changes for the following, in priority order:
- Add the visual overview (inline comment on the "Selecting functions" heading). This is the main change I'd like to see.
- Fix the imprecise selection-order sentence (inline comment).
- Make the two fragile heading anchors robust (inline comments).
- Confirm the docs build. Only the "Auto Label" check has reported on this PR, so no CI job has built the book. Please run
mdbook build docslocally and confirm there are no new warnings and no broken links, or get the docs workflow approved to run. - Optional, not blocking: the page is now ~790 lines. Consider a follow-up that splits it into overview, argument generation, and soundness pages.
I'll re-review once these are addressed.
| Complete - 0 successfully verified functions, 1 failures, 1 total. | ||
| ``` | ||
|
|
||
| ## Selecting functions |
There was a problem hiding this comment.
Add a visual overview here. The headings follow the structure from #4979, but the three-phase diagram from that issue is not in the page; only the contributor architecture diagram at the bottom is. Readers who want the big picture first have nothing to anchor on.
Please add a short "How it works" section before this heading (after the Example) with the SELECT / GENERATE / VERIFY & REPORT diagram, and link each box to its section here (#selecting-functions, #generating-harnesses, #verifying-and-reading-results, #soundness-caveats). The diagram is in #4979 (comment). Update the arguments/assumptions/checks lists in it only where this page differs.
There was a problem hiding this comment.
Added a "How it works" section after the Example with the diagram from #4979. The box titles link to their sections (#selecting-functions, #generating-harnesses, #verifying-and-reading-results, #soundness-caveats, and the architecture section). To make the boxes clickable while keeping the ASCII layout, the diagram is a raw <pre><code class="language-text"> block, so it renders the same as a regular text code block.
I compared the arguments/assumptions/checks lists against the page and every item has a matching section. The only change is "mined inv. (ctor)", since the page documents mined invariants as applying only under --constructor-args. The rest of the diagram is unchanged from the issue.
|
|
||
| Autoharness considers functions defined in the crate being verified, including free functions, inherent methods, trait-implementation methods, and trait methods with default implementations. Functions from dependencies are not candidates. For each eligible function, Kani selects one automatic harness; functions that fail a selection check are skipped. Kani reports selected functions and visible skip reasons before verification, unless `--quiet` is specified. | ||
|
|
||
| Selection first attempts to instantiate generic functions, then applies the include and exclude patterns, and finally checks whether the function's arguments can be generated. The details of argument generation are described in [Generating harnesses](#generating-harnesses). |
There was a problem hiding this comment.
This ordering is slightly imprecise. In automatic_harness_partition (kani-compiler/src/kani_middle/codegen_units.rs) the checks run in this order:
- Kani instrumentation functions (
KaniImpl) and compile-time-only functions (Comptime), both before generic instantiation; - generic instantiation (
GenericFn); NoBody, then proof harnesses and Kani trait impls (KaniImpl), then the include/exclude filter (UserFilter), then unsupported variadic;- argument support (
MissingArbitraryImpl/RequiresBoundedArguments).
Either list this order, or soften the sentence (for example "Selection roughly proceeds as follows") so it does not claim instantiation is the first check. The skip-reason table could also say that the first failed check wins, which it already does in the paragraph above the table.
There was a problem hiding this comment.
Thanks, good catch. In 3186465 the paragraph now lists the checks in the order automatic_harness_partition runs them: Kani-internal and compile-time-only functions before generic instantiation, then the body, proof-harness/Kani trait impl, include/exclude and variadic checks, and finally argument support. The skip-reason table intro now also says its rows follow that order (they already did). I also noted that compile-time-only functions, like generic ones, are reported regardless of the filters.
| that use bounded values are marked **"(bounded)"** in the summary table, and a note after the | ||
| #### Opt-in under-approximations | ||
|
|
||
| ##### Bounded Arguments (opt-in: `--bounded-arguments`) |
There was a problem hiding this comment.
The link target for this heading, #bounded-arguments-opt-in---bounded-arguments, is used in several places (skip reasons, Default settings, Formatting, Soundness caveats). It works today, but it depends on how mdBook turns the punctuation in this heading into an id, so a small mdBook change would break all of those links at once. Consider a plain heading such as ##### Bounded arguments (--bounded-arguments) or ##### Bounded arguments, and update the links. (This also avoids the near-duplicate with the ### Bounded arguments heading under Soundness caveats, which is currently disambiguated only by the suffix.)
There was a problem hiding this comment.
Renamed to "Bounded argument generation" (id bounded-argument-generation) and updated all five links in 3186465. I didn't use either suggested name because each runs into one of the other concerns: "Bounded arguments (--bounded-arguments)" has a plain-text "--", so its id would depend on smart punctuation like the constructor heading, and plain "Bounded arguments" would take the soundness caveat's id and push that one to bounded-arguments-1. The flag is still named in the section's first paragraph.
| `kani::any`), so they are generated by default and are unaffected by `--bounded-arguments`. If the | ||
| implementing type implements [`Invariant`](#type-safety-invariants), the generated value is assumed | ||
| to satisfy it, as for any other automatic harness. | ||
| ##### Constructor-based generation (--constructor-args) |
There was a problem hiding this comment.
Same concern as the Bounded Arguments heading: #constructor-based-generation-constructor-args only resolves because of how mdBook's smart punctuation treats the -- in a plain-text heading (Copilot flagged this and the explanation in the thread is right for the current build). Wrapping the flag in backticks, as --check-invariants already is, or dropping the flag from the heading text, would make the id independent of that setting.
There was a problem hiding this comment.
Dropped the flag from the heading, so it is now "Constructor-based generation" (id constructor-based-generation), and updated both links in 3186465. "Listing functions (--list)" had the same issue (no links yet), so I dropped the flag there too.
Describe the selection checks in the order automatic_harness_partition runs them: Kani-internal and compile-time-only functions are rejected before generic instantiation, followed by the body, proof-harness, include/exclude, variadic and argument checks. State that the skip-reason table follows this order. Rename the bounded-arguments, constructor-args and --list headings so their ids no longer depend on how mdBook treats "--" and other punctuation, and update all links. This also removes the near-duplicate with the "Bounded arguments" soundness caveat.
Add the three-phase SELECT / GENERATE / VERIFY & REPORT diagram from model-checking#4979 before "Selecting functions", with links to the section for each phase, the soundness caveats and the architecture section. The diagram marks mined invariants as applying only under --constructor-args, as the page describes.
… docs/autoharness-restructure
Render the diagram as a raw HTML <pre> block so the box titles can link to their sections, as asked in review, and drop the separate link list. Restore the diagram text to the model-checking#4979 original apart from the mined invariants line.
|
Thanks for the careful review! All items are addressed:
|

Resolves #4979.
This rewrites the autoharness reference page (
docs/src/reference/experimental/autoharness.md) to match the implementation onmain, following the structure proposed in #4979: selecting functions, generating harnesses (arguments, assumptions, checks), verifying and reading results, soundness caveats, and an architecture overview for contributors.How the items in #4979 are addressed:
Vec: new "Primitive slices and Vec" section; "Bounded Arguments" and "Arguments Implementing Arbitrary" now distinguish the primitive unbounded models from the bounded generators.Fn-bound generics: "Generic Functions" (instantiation) and "Fn-bound closures" (the model and its limits).Box/Rc/Arc: new "Box, Rc and Arc" section.--stdin "Usage",--list --formatin "Listing functions (--list)", and the bound flags in "Bounded Arguments".The soundness caveats are now collected in one section: single monomorphization, aliasing between arguments, bounded arguments, vacuous constructors, and the mined-invariant heuristic. That section also has an "Other limitations" subsection that is not in the proposed structure. It lists the under-approximations that apply by default without a marker in the summary table. Happy to drop or move it if you prefer.
While writing the page we found and reported two issues, both described in the caveats:
Fn-bound closure models have no side effects, so callbacks that modify shared state are not covered.--constructor-args, mined-invariant filtering of a type with its ownArbitraryimplementation is not marked "(ctor)".Other fixes:
Arbitrary.Notes for review:
--alias-arguments.Contributors:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.