Repository navigation
Judge whether a conversation achieved its targets, and what it cost - #87
Merged
Merged
Conversation
wu-sheng
force-pushed
the
conversation-judge
branch
2 times, most recently
from
October 22, 2026 03:09
bfcb4ba to
f4b99dd
Compare
Member
Author
Example: what the judge printsA real run on a local Claude Code conversation that reviewed apache/skywalking#13943, with the build of this pull request. The conversation has 5 main talks, and its longest one started child agents. The evaluators are two agent CLIs, named in judge:
evaluators:
opus: {kind: cli, program: claude, model: claude-opus-5-5, args: ["--effort", "high"]}
sonnet: {kind: cli, program: claude, model: claude-sonnet-5-5, args: ["--effort", "high"]}The price file used here lists neither model, so costs are counted in tokens, and the cost each CLI reported is kept beside them. What the conversation consumed, with no model calledA judge runEach stage goes to the evaluator the plan names; here the sub-tasks go to sonnet. What a stage consumes is printed while it runs and when it ends. Two stages are partial, and each says why: one sub-task decision had no valid evidence and was weakened, and one talk of a target was not labeled. The resultEach target the person set, with its outcome, its cost and what its work was; then the sub-tasks, and the two scores, which are never combined. t1 is the request the person corrected right after typing it, so it is replaced and not scored. Every run of the conversationThe run took 3.5 minutes. Writing this example showed two printing faults, both fixed in the commit: a share of a call was cut to two significant digits (60.5 calls printed as 60), and the score line counted a replaced target that the score leaves out. |
For each target the person set in a conversation, the judge says whether it was achieved and how efficiently the time and tokens were spent. asz measure prints what a conversation's model calls consumed: per talk, with the child agents each one started, at any depth, priced where an optional price file names the model. It calls no model. asz judge runs a plan of stages over one conversation, or a window of it, and writes one verified file per stage: - ledger, computed: every model call counted once, and its owner. - targets: the targets the person set, and what each input did to them. - achievement: whether each target was achieved, criterion by criterion, with citations. - subtasks: for each sub-agent, its outcome, whether its report was true, whether it was used, and whether giving it the task was reasonable. A task given wrong and an untrue report acted on are charged to whoever caused them. - efficiency: which parts of the agent's own work were avoidable. - close, computed: each target's final statement, costs, and two scores, achievement and efficiency, which are never combined. An evaluator reads the conversation only through a bounded navigation, served as an MCP server to agent CLIs and in process to model APIs. Every answer fits a page, every call has a reading budget, and nothing past the run's head or window is shown. asz checks every answer against the record: a decision with no valid evidence falls back to the value that claims nothing, and the stage file says what was weakened. What judging consumes is counted per model request as the provider reported it, never estimated. Judge runs go to the OTLP receiver beside the conversation files. They are removed with their session, and scenario removal waits until they are sent. The sj/1 file format is documented in docs/en/formats/session-judgment.md.
wu-sheng
force-pushed
the
conversation-judge
branch
from
October 22, 2026 06:00
f4b99dd to
1c5bd4d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds the conversation judge for 0.7.0. For each target the person set in a conversation, it says
whether the target was achieved and how efficiently the time and tokens were spent.
What it adds
asz measureprints what a conversation's model calls consumed: per talk, with the child agentseach one started, at any depth, priced where an optional price file names the model. It calls no model.
asz judgeruns a plan of stages over one conversation, or a window of it, and writes oneverified file per stage:
ledger, computed: every model call counted once, and its owner.targets: the targets the person set, and what each input did to them.achievement: whether each target was achieved, criterion by criterion, with citations.subtasks: for each sub-agent, its outcome, whether its report was true, whether it was used, andwhether giving it the task was reasonable. A task given wrong (
misdirected) and an untrue reportacted on (
misled) are charged to whoever caused them.efficiency: which parts of the agent's own work were avoidable.close, computed: each target's final statement, costs, and two scores, achievement andefficiency, which are never combined.
agent CLIs and in process to model APIs. Every answer fits a page, every call has a reading budget,
and nothing past the run's head or window is shown.
value that claims nothing, and the stage file says what was weakened.
printed while a stage runs and when it ends.
session, and scenario removal waits until they are sent.
sj/1file format is documented indocs/en/formats/session-judgment.md.Docs
docs/en/concepts-and-designs/conversation-judge.md: the design, with the measurements behind it.docs/en/formats/session-judgment.md: the stage files.docs/en/setup/judge.md,configuration.md,command-line.md: how to set it up and run it.Tests
make checkpasses: vet, lint, license headers, dependency licenses, and every test under the racedetector. The runner is tested end to end with stand-in evaluators.
Every guard of the judge was made false in turn, and in the answer checks, the close file and the stage
file reader and writer, each part of a condition as well; each change was compiled before the tests ran.
A test fails for every one of them, except a few conditions implied by another and guards that no
document asz builds can reach, which are kept against a malformed one. The operating system failure paths
of the evaluators and of the judgment storage are not tested yet; they are listed below.
Follow-up
These failure paths return an operating system or transport error, and a test needs a way to make the
operation fail. Each test should check that the error is returned, and for an evaluator that no request is sent.
cliExecutor.Run: writingprompt.md, the answer schema and the MCP configuration;making Kiro's agent directory and writing its
asz-judge.json.apiExecutor.RunandapiExecutor.post: writingprompt.md; building the request andreading the response body.
credentialsandcallBedrock: the AWS command-line tool failing, or printing credentials thatdo not decode or lack an access key ID; building the request, the transport, and reading the body.
JudgmentRuns,ReserveRun,PublishStageandMakeWorkDir: reading directories,taking the root lock, and making the run's directories. Their retirement checks are already tested.