Issues

• Proposal update: Independent black-box evaluation of consumer AI under deployment opacity — historical v0.1 retained

  openai/evals#1839 · Filter by author TheMockingjaysings opened on Sep 26, 2026
  5

• naughty_strings_graded scores judgement rather than generation, on a scale whose optimum is 0.5, reported as accuracy

  openai/evals#1837 · Filter by author shaurya416 opened on Sep 23, 2026
  2

• [Architecture Proposal] Hardware-Gated Containment Framework (Genesis Protocol V5.0) for Recursive Self-Improvement

  openai/evals#1833 · Filter by author guienjavar-a11y opened on Sep 13, 2026
  2

• Proposal: harness-level failure attribution in eval reports (model vs harness vs eval definition)

  openai/evals#1832 · Filter by author UniversePeak opened on Sep 13, 2026
  1

• Proposal: RES actor-indexing behavioral eval (20 samples, no custom code)

  openai/evals#1829 · Filter by author NeonInsight opened on Sep 9, 2026

• Proposal: deterministic eval for agent action-boundary violations

  openai/evals#1827 · Filter by author UniversePeak opened on Sep 5, 2026

• False-positive "distillation" ban on $200/mo Pro subscriber — automated appeal rejected by bot, requesting human review from Trust & Safety

  bug
  openai/evals#1826 · Filter by author stpblgm-crypto opened on Sep 5, 2026
  5

• Proposal: detect tool-trajectory regressions before deployment

  openai/evals#1825 · Filter by author UniversePeak opened on Sep 5, 2026
  1

• Proposal: Shared Sparse Latent Neural Bus for Local ↔ Cloud Models

  openai/evals#1824 · Filter by author nikimagic opened on Sep 5, 2026

• HttpRecorder silently discards logged eval events; HTTP error responses never trigger the fallback at any failure rate

  openai/evals#1819 · Filter by author AUTHENSOR opened on Aug 29, 2026

• API retry helper retries forever (no max_tries/max_time) and the error-detection branch in its callers is dead code

  openai/evals#1818 · Filter by author AUTHENSOR opened on Aug 29, 2026

• [Bug] Context/Token Explosion in Geometric Script Generation (CadQuery/CalculiX)

  bug
  openai/evals#1817 · Filter by author kumar1441 opened on Aug 26, 2026

• LangChain completion wrappers drop runtime kwargs

  openai/evals#1811 · Filter by author sylvesterkaczmarek opened on Aug 21, 2026

• Steganography does not flag missing required protocol fields

  openai/evals#1810 · Filter by author sylvesterkaczmarek opened on Aug 21, 2026

• SchellingPoint mutates process-wide random state

  openai/evals#1809 · Filter by author sylvesterkaczmarek opened on Aug 21, 2026

• MakeMeSay get_content fails on its documented dict response type

  openai/evals#1805 · Filter by author sylvesterkaczmarek opened on Aug 21, 2026

• ML Agent Bench crashes when an action omits Action Input

  openai/evals#1804 · Filter by author sylvesterkaczmarek opened on Aug 21, 2026

• Ballots role reversal rewrites system instructions as assistant messages

  openai/evals#1803 · Filter by author sylvesterkaczmarek opened on Aug 21, 2026

• Post-hoc control recall counts undefined no-control samples after parse failures

  openai/evals#1799 · Filter by author sylvesterkaczmarek opened on Aug 21, 2026

• Empty prompt lists are misclassified as chat prompts

  openai/evals#1798 · Filter by author sylvesterkaczmarek opened on Aug 21, 2026
  2

• LangChain math adapter can corrupt answers while removing Answer prefix

  openai/evals#1797 · Filter by author sylvesterkaczmarek opened on Aug 21, 2026

• Tool conversation runner overcounts exhausted runs by one turn

  openai/evals#1793 · Filter by author sylvesterkaczmarek opened on Aug 21, 2026

• TrackTheStat caps a perfect 300-turn run at 299

  openai/evals#1792 · Filter by author sylvesterkaczmarek opened on Aug 21, 2026

• BluffEval can leave registry logging disabled after failed solver lookup

  openai/evals#1791 · Filter by author sylvesterkaczmarek opened on Aug 21, 2026

• FunctionDeduction parser merges comma-separated integers without spaces

  openai/evals#1787 · Filter by author sylvesterkaczmarek opened on Aug 21, 2026