Skip to content

Commit c7bef36

Browse files
Tomkessclaude
andcommitted
fix(gooddata-eval): report whole-conversation latency and ungated cost
Third structural finding from #1799's review, applied here. latency_sec used run.turn_wall_clock_sec, which is the goal turn alone and excludes the clarification turns that got there -- understating the item's real elapsed cost on exactly the runs where it matters. It now prefers pt.latency and falls back to the goal turn, which is what 7 of the 8 existing kinds already do; kda_skill is the outlier and documents its own reason, and this copied it without re-checking. cost_usd was gated on ev.triggered, so a run that answered without ever reaching the tool reported no cost despite having spent tokens. The gate is gone. 806 passed, lint and format clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 273b842 commit c7bef36

1 file changed

Lines changed: 9 additions & 2 deletions

File tree

‎packages/gooddata-eval/src/gooddata_eval/core/agentic/forecasting.py‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -526,8 +526,15 @@ def _write_scores(ctx: RunTraceContext) -> None:
526526
ctx.quality(
527527
tid,
528528
strict_checks=strict_checks,
529-
latency_sec=run.turn_wall_clock_sec,
530-
cost_usd=pt.total_cost if pt and ev.triggered else None,
529+
# pt.latency covers the whole conversation, which is the item's real
530+
# elapsed cost when the agent needed clarification turns to get
531+
# there; turn_wall_clock_sec (the goal turn alone) is the fallback.
532+
# This is what 7 of the 8 existing kinds do -- kda_skill is the
533+
# outlier and documents its own reason. Cost is not gated on
534+
# ev.triggered: a run that answered without ever reaching the tool
535+
# still spent tokens, and hiding that understates what the item cost.
536+
latency_sec=pt.latency if pt else run.turn_wall_clock_sec,
537+
cost_usd=pt.total_cost if pt else None,
531538
)
532539

533540
# Before the pass@K raise: a failing item's scores are the ones worth having.

0 commit comments

Comments
 (0)