Issues

• evals: dead editable key, and temp paths make rendered prompts non-reproducible

  heibench/partspec#419 · Filter by author CameronBrooks11 opened on Oct 19, 2026

• evals: the freeze hash is non-recursive, so a shadowing module can change the contract unchecked

  heibench/partspec#418 · Filter by author CameronBrooks11 opened on Oct 19, 2026

• evals: contract tampering on the final turn is scored failed, not gamed

  heibench/partspec#417 · Filter by author CameronBrooks11 opened on Oct 19, 2026

• evals: the regressed outcome is never assigned, so the bucket reports zero by construction

  heibench/partspec#416 · Filter by author CameronBrooks11 opened on Oct 19, 2026

• evals: the oracle is partspec, so it cannot grade an experiment where partspec is the treatment

  heibench/partspec#415 · Filter by author CameronBrooks11 opened on Oct 19, 2026

• evals: run.py has no tests and no CI, so a regression surfaces only in a paid run

  heibench/partspec#414 · Filter by author CameronBrooks11 opened on Oct 19, 2026

• evals: fleet-01 is cited in the changelog but its evidence is untracked, so the numbers cannot be checked

  heibench/partspec#413 · Filter by author CameronBrooks11 opened on Oct 19, 2026

• evals: five repair cases cap every claim; grow to ten or more from real fix history

  heibench/partspec#412 · Filter by author CameronBrooks11 opened on Oct 19, 2026

• evals: add a no-tool arm, and decide its stopping condition before any paid run

  heibench/partspec#411 · Filter by author CameronBrooks11 opened on Oct 19, 2026

• evals: arms are four hard-coded string comparisons, and a new arm name silently runs everywhere

  heibench/partspec#410 · Filter by author CameronBrooks11 opened on Oct 19, 2026

• evals: the harness records no token or cost data, so the tool's price is unmeasurable

  heibench/partspec#409 · Filter by author CameronBrooks11 opened on Oct 19, 2026

• The release audit, the adoption-study method and a handful of engine traps are recorded nowhere in the repo

  heibench/partspec#405 · Filter by author CameronBrooks11 opened on Oct 5, 2026

• OpenSCAD render needs a display for its views and drops the section with them, though raster.py can draw both from the STL

  heibench/partspec#401 · Filter by author CameronBrooks11 opened on Sep 27, 2026

• A model that fails an OpenSCAD assert() renders nothing, and empty() passes on it

  heibench/partspec#400 · Filter by author CameronBrooks11 opened on Sep 27, 2026
  1

• The exemplar replay ignores order, indent, and four of five console blocks

  bug
  heibench/partspec#379 · Filter by author CameronBrooks11 opened on Sep 3, 2026
  1

• Geometry vanished from a failed range makes a part too small; a two-sided envelope is what catches it

  bugsilence-defect
  heibench/partspec#338 · Filter by author CameronBrooks11 opened on Aug 24, 2026
  2

• A defaulted scalar's .csg is byte-identical to a correct part's, so no .csg rule can decide #308

  bugsilence-defect
  heibench/partspec#332 · Filter by author CameronBrooks11 opened on Aug 24, 2026
  2

• Record whether the kernel retains a zero-thickness result, as provenance only

  enhancement
  heibench/partspec#314 · Filter by author CameronBrooks11 opened on Aug 22, 2026
  2

• A dimension defaulted from undef passes at exit 0, and no channel can refuse it without refusing correct parts

  bugsilence-defect
  heibench/partspec#308 · Filter by author CameronBrooks11 opened on Aug 22, 2026
  1

• No way to check an existing mesh file; every source must be built from CAD

  heibench/partspec#275 · Filter by author CameronBrooks11 opened on Aug 22, 2026

• Chained exceptions bypass traceback filtering entirely; per-segment filtering was deferred by #197

  heibench/partspec#203 · Filter by author CameronBrooks11 opened on Aug 14, 2026