Skip to content

Add generation/thinking truncation metrics for generative tasks - #11

Open
Jeronymous wants to merge 1 commit into
mainfrom
feature/generation_truncated
Open

Jeronymous wants to merge 1 commit into
mainfrom
feature/generation_truncated

Conversation

@Jeronymous

Copy link
Copy Markdown
Member

Report how often a model's generation is cut off by reaching a token budget, distinguishing the answer from a thinking model's reasoning.

  • ModelResponse gains per-generation output_truncated (answer / whole generation) and reasoning_truncated (two-phase reasoning phase) flags.
  • vLLM and SGLang set them from the native finish reason; the two-phase thinking path sets them from the per-phase budgets, so they work for reasoning models on any backend.
  • New Metrics.generation_truncated and Metrics.thinking_truncation report a rate (mean) and a count (sum).
  • Both are added automatically to every generative task via LightevalTaskConfig, so no task needs editing by hand.

Not covered (reported as 0): non-thinking transformers/sglang single-phase stop-sequence stops have no per-sample finish reason, and remote endpoints.

@Jeronymous
Jeronymous requested a review from Oligou November 3, 2026 13:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant