Skip to content

Hold the paused reply once in ToolCallLimitExceededException history - #7213

Open
Ridzzz0Alam wants to merge 1 commit into
spring-projects:mainfrom
Ridzzz0Alam:fix-limit-breach-history-on-resume
Open

Ridzzz0Alam wants to merge 1 commit into
spring-projects:mainfrom
Ridzzz0Alam:fix-limit-breach-history-on-resume

Conversation

@Ridzzz0Alam

Copy link
Copy Markdown

When an application resumes a paused tool call batch with ToolCallingManager#executeToolCalls(Prompt, ChatResponse, ToolCallDecisions) the prompt may already end with the paused reply. This happens for example when the application passes its whole conversation. DefaultToolCallingManager drops that trailing reply before it builds the conversation history so the reply appears only once.

The branch that throws ToolCallLimitExceededException did not do the same. It built the partial history from the raw prompt.getInstructions() so the paused reply appeared twice:

[USER, ASSISTANT (paused reply), ASSISTANT (paused reply), TOOL (partial results)]

An application that catches the exception and sends getPartialToolExecutionResult().conversationHistory() back to the model gets an error from the provider. The first copy of the reply carries tool calls without results.

Reproducer

ToolCallingManager manager = ToolCallingManager.builder().maxCallsPerTool(1).build();
AssistantMessage paused = /* reply with two calls to the same tool, ids c1 and c2 */;
Prompt prompt = new Prompt(List.of(new UserMessage("Book VX205"), paused), optionsWithTools);

manager.executeToolCalls(prompt, new ChatResponse(List.of(new Generation(paused))),
		ToolCallDecisions.of(ToolCallDecision.allow("c1"), ToolCallDecision.allow("c2")));
// ToolCallLimitExceededException: partial history holds the paused reply twice

Fix

Compute the de-duplicated previous messages before any call runs and pass them to executeToolCall. The regular history and the partial history of a limit breach are then built from the same messages. The method executeToolCall is private so there is no API change.

How tested

Added promptEndingWithTheReplyHoldsItOnceInThePartialHistoryOfLimitBreach to DefaultToolCallingManagerDecisionsTests. The test fails without the fix (expected size 3 but was 4) and passes with it.

./mvnw -pl spring-ai-model clean package -Dmaven.build.cache.enabled=false

The spring-ai-model module on JDK 17.0.19 ran 1037 tests with 0 failures.

The async tool calling proposal in #4756 builds its own history on the limit path. It may need the same handling if it is picked up again.

When an application resumes a paused tool call batch, the prompt may
already end with the paused reply. `DefaultToolCallingManager` drops
that trailing reply before it builds the conversation history, so the
reply appears once. The branch that throws
`ToolCallLimitExceededException` still built its partial history from
the raw prompt instructions, so the reply appeared twice, followed by
the partial tool results. An application that sends that history back
to the model gets an error, because the first copy of the reply has
tool calls without results.

Compute the de-duplicated previous messages before the calls run and
use them for both the regular and the partial history.

Signed-off-by: Ridzzz0Alam <ridwan0gts@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants