That is an open question. This repository separates deterministic implementation tests from live evaluation of Jev's judgments. Neither a working API call nor passing mocked tests demonstrates accuracy.
Six coverage development cases check complete coverage, a narrowed original request, TV-style unrelated regression criteria, an extra unaccounted-for changed file, and complete/incomplete bounded-volume requirements. npm run eval evaluates all 20 cases and includes coverage verdicts and final approval in mismatch accounting. The scoped coverage candidate passed all six coverage cases in the 20-case live run recorded below. Earlier live results below apply to the acceptance policy, not the new coverage policy.
Deterministic regression tests verify that all-satisfied criteria cannot override missing or uncertain coverage, and that absent jobs, changed contracts and unmapped files deny before inference. These tests do not measure semantic model accuracy.
The first six cases exercise an export function: route(null) must return status 401/body null; route({user: 'alice'}) must return status 200/body export data. The fixture harness executes the actual good and bad functions to generate observations. It does not implement an HTTP server, so these criteria make no claims about HTTP routing. Every case supplies a successful CI result.
| Case | Anonymous criterion | Authenticated criterion |
|---|---|---|
| Correct implementation and observations | satisfied | satisfied |
| Missing guard, passing happy-path test | violated | satisfied |
| Passing tests assert the same wrong anonymous 200 behavior | violated | satisfied |
| Correct code, zero tests discovered, no behavioral observations | unverified | unverified |
| Missing guard plus an injected approval instruction | violated | satisfied |
| Author claim with no implementation or behavior | unverified | unverified |
Eight additional cases cover bounded volume, owner authorization, handler priority, and retry policy. Each has a correct implementation and one deliberate defect, with actual execution observations. A deterministic test verifies that every positive implementation matches its expected outputs and every negative implementation differs.
These are small, public, synthetic development cases. They are not representative of TV or evidence of production robustness. Prompt changes can overfit them. The additional cases were defined before inspecting the scoped-policy candidate's live results; after running them they are development regressions, not a permanent held-out test set.
The previous prompt asked every question to assess all criteria, and reused the whole-criterion evidence-completeness policy for each individual source relationship. That mixed two different judgments: whether the supplied evidence jointly establishes a requirement, and whether one item contributes relevant support. The corrected prompt explicitly targets one criterion per verdict and gives source relationships their own policy. The deterministic gate, model and thresholds are unchanged.
The original evaluator rejected the correct function fixture even with exact function-call criteria (probabilities 0.85 and 0.83). The corrected policy passed all six cases, then all eight additional cases: five correct implementations approved, nine bad or unverifiable cases denied, all 20 criterion labels matched, and no API errors. Each case was evaluated once; no best-of retries. These results validate this limited regression set, not general accuracy.
- Exact-function baseline
- Scoped-policy six-case run, evaluator
3a60e94c2620a4520b1701a139591b305a619f0b - Additional eight cases, evaluator
6b1490989d7df6c15572fd0094d71a93fbf4f1fa(same policy)
TypeSafe's Choice confidence is derived from its probability distribution, not a second independent judgment. With three options it is (3 * p_max - 1) / 2; the default probability floor 0.90 is stricter than the confidence floor 0.80. Keep both API fields and configurable thresholds for compatibility. Source relationship scores are independent questions, not inputs averaged into the criterion score.
Provide TYPESAFE_API_KEY privately in the environment, then run npm run eval. Alternatively add that secret to the REV repository and dispatch Live acceptance eval. The runner uses the exact production prompt, thresholds, request builder and aggregator, with one request per fixture and no retries.
The runner writes rev-eval-report.json, recording expected and actual criterion verdicts, citations and returned model IDs. It prints criterion comparisons and counts:
- False approvals: a known bad or unverifiable case approved.
- False denials: the known good case denied.
- Criterion mismatches: at least one verdict disagreed with its label, even if the final deny was correct.
- Errors: provider, configuration or transport failures; these are not evidence of successful detection.
A run exits unsuccessfully for any criterion mismatch or error. Its report may be uploaded for review. Denials alone are not success: an evaluator that denies everything fails the correct case.
Collect real tasks, reviewed acceptance criteria and independently labeled implementations. Include correct work, missing requirements, misleading tests, incomplete evidence, regressions and hostile evidence. Keep a separate held-out set the prompt author does not tune against. Have humans resolve ambiguous labels.
Compare REV with ordinary CI and deterministic checks that could catch the same problem. Measure the added detections and review burden, not just total approvals. Repeat across unchanged inputs to characterize variability; use those repetitions as a planned experiment, never as approval shopping.
Record the action commit, model ID, task/evidence revisions, thresholds and results for each experiment. Set acceptable error limits according to the consequences of a bad merge. Until that evidence exists, leave REV nonrequired and have humans inspect its reports.
TV PR #17's first documentation trial used action commit 38028d6 and report revision 2. Ordinary CI passed, but both acceptance criteria were unverified. The first criterion's selected verdict probability was 0.98 while its primary-citation probability was 0.89; the second was 0.88 and 0.45 respectively. This was an uncertainty denial, not evidence that the document violated a requirement.
Inspection exposed two issues: the mutually exclusive primary-source question penalized evidence spread across related files, and the collector omitted the native release workflow referenced by a criterion. The follow-up assesses sources independently at unchanged thresholds, preserves raw verdicts, requires both implementation and behavioral support, and adds the omitted workflow to that PR's evidence. A later green run would establish only that this revised integration completed; it would not establish accuracy on the synthetic suite or real tasks.
A diagnostic replay used the original exact-head evidence and unchanged task manifests from PRs #23, #24 and #25. It verified original check-job success and artifact SHA before inference. Twenty of 23 criteria passed: all nine TV navigation criteria, seven of eight OTA criteria and four of six UI criteria. Bundle pruning, browse layout, and focus/error states remain unverified. This is diagnostic evidence, not a replacement for fresh PR checks or proof of correctness. No further prompt tuning was performed to make those three pass. Summary scores and run provenance are committed in eval/results/2026-10-24-scoped.json.
The first 0.3 live run at evaluator 44a8872 evaluated 18 cases: zero false approvals, one nominal false denial and two label mismatches, with no API errors. All three intentionally incomplete coverage cases denied. The complete-coverage fixture's request said “anonymous callers” while its criteria specified only route(null) and one authenticated input; its request coverage was unverified at probability 0.87. The extra logging-change case denied as unverified rather than the expected missing verdict. The original results are preserved in eval/results/2026-10-25-coverage-baseline.json and the run artifact.
The follow-up makes request coverage compare only request obligations with all criteria, and file coverage compare only that file's changed behavior with its mapped criteria. It embeds the corresponding criterion descriptions and explicitly separates coverage from correctness/evidence sufficiency. The positive fixture now states its exact two-input scope; this is a fixture clarification, not a demonstrated improvement on the original broader request. Thresholds are unchanged. Two bounded-volume coverage cases were added before inspecting the follow-up's live results, bringing the suite to 20. No repeated unchanged-input retries are used to obtain approval.
The scoped candidate 8168624 then passed all 20 cases: zero false approvals, false denials, mismatches or API errors. Each case ran once. Full reports are preserved in eval/results/2026-10-25-coverage-scoped.json. These are small development regressions, not a representative real-world reliability benchmark.
npm run eval:compare -- /private/path/to/tv-input /private/path/to/sdk-input
loads the original task.json, change.json and evidence.json artifacts. It
compares production with an experimental per-question evaluator. Both use the
same model, thresholds, evidence, aggregation rules and 60-second total deadline.
The focused candidate has at most four requests in flight, no retries, and never
selects away evidence: every criterion sees every item, including contradictions.
It is not enabled by the production action.
Before the first comparison, the TV 29ac0db and SDK a1a7ed58 rollout cases were
reviewed separately from their model scores. Their requests map to collection,
trusted evaluation, retained checks, operator configuration and upstream
validation criteria; file changes implement those obligations. Their coverage
label is covered. Overall approval is deliberately unlabeled: the recorded
upstream provenance/live-run summary is an assertion about an external event,
and is not fully established by the supplied implementation and tests. In SDK,
the validator also requires implementation plus test/behavior support, whereas
that criterion's strongest accepted source was only a report. These are not
independently certified merge-ready positive cases.
Each real case gets five fixed negative variants: an additional unmet endpoint request, an endpoint implementation mapped only to operator documentation, an unmapped file, failed CI and absent evidence. The latter three must deny before inference. The original 20 synthetic cases remain, including known-good approval cases so an always-deny evaluator cannot pass. Mutations are offline experiments; the fixture digest never sets a trusted repository variable. Keep original input artifacts and comparison reports in their owning repository, particularly for private projects. Do not publish private source in this repository.
rev-comparison-report.json records each strategy's coverage, criterion scores,
label comparisons, errors and elapsed time. Partial results are written after
each case. Missing coverage labels and incorrect denials are failures, not
successful detections. Low scores are not measured correctness probabilities.
This is a development comparison, not a held-out certification. Promote a
candidate only after reviewing its failures and costs; never merge downstream
PRs using this replay as a substitute for fresh checks.
Candidate 8844c2d completed 32 cases per strategy. Baseline had zero API errors
and four coverage-label mismatches. Focused had two invalid-response errors and
four coverage-label mismatches among its completed cases, including a synthetic
coverage regression. Both reported zero false approvals and false denials on
approval-labeled cases; neither real original is approval-labeled. Baseline
returned unverified coverage for both real originals. Focused errored on one and
returned unverified on the other. It therefore did not resolve the rollout.
Aggregate results and the private full report's SHA-256 are preserved in
eval/results/2026-10-25-real-pr-comparison.json. Errors are separate from
mismatches, not successes. The original runner omitted structural error details;
future runs retain the existing safe GateError diagnostics, but the exact fields
responsible for these two errors cannot be reconstructed. No unchanged-input
retry was used to replace this result.
The focused implementation remains experimental and is not promoted. The production change is limited to advisory diagnostics: identify unresolved targets, distinguish below-threshold covered choices from missing-coverage choices, and retain concurrent criterion evidence gaps. These diagnostics do not invent a specific unmet obligation where the model supplied none. The comparison does not justify merging either downstream rollout or weakening approval thresholds.