Repository navigation
feat: run sessions on a local llama.cpp server - #365
Merged
Merged
Conversation
A new `llamacpp` backend drives a `llama-server` the user runs, over its
OpenAI-compatible API: OpenAIProvider under its own id, so it sits beside
a cloud `openai` gateway instead of replacing it. Off by default.
Built from a first-cut patch, then run against a real llama-server
(Qwen2.5-7B-Instruct), which surfaced four things the first cut missed:
- Every turn failed with a 500 when MCP servers were mounted. llama.cpp
compiles tool schemas into a grammar and rejects an annotation-only
subschema like `{ "description": "any value" }`, which MCP servers emit
for untyped parameters. strictJsonSchema spells "any value" out as every
JSON type, the same constraint in a form it accepts, and every tool gets
a description because Qwen-style chat templates require one.
- External MCP tools overflowed the model before the user typed: six
imported servers came to ~42k tokens of tool definitions against a 32k
context. External servers are now opt-in per server
(`providers.llamacpp.mcpServers`); built-in memory tools stay on.
- The context window was a cloud-sized 128k guess. The server's real
per-request window is read from `/props` after every turn, and the
pre-first-turn floor is llama-server's own default of 4096, so seeds on
a switch or fork, the context meter and auto-rotate size from what the
server can actually hold.
- An over-context error from older llama-server builds arrives as a line
the OpenAI SDK drops, so the turn "succeeded" with no answer. A stream
that ends before its first chunk now fails the turn with the likely
cause.
Also: the local model list is no longer filtered to gpt-*/o* ids (and an
unreachable server lists nothing rather than GPT models), errors are
prefixed with the backend's own name, and the session tenant is passed so
OAuth MCP servers resolve credentials.
User guide in docs/local-models.md; the CLAUDE.md section is the
maintainer summary.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
KunalJavelin
approved these changes
Oct 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Adds a
llamacppbackend so a codeoid session can run on an open-weight model served on your own machine by llama.cpp'sllama-server— no API key, no per-token cost, nothing leaving the machine except what your MCP servers send.It is off by default. Turning it on is two steps, documented in docs/local-models.md:
llama-server -m model.gguf --jinja -ngl 99 -c 32768.providers.llamacpp.enabled: true(andbaseUrlif not on port 8080), restart the daemon, and create a session with--provider llamacpp.It is a separate backend from
openai, so a local server and a cloud OpenAI-compatible gateway can be enabled side by side.What testing against a real server changed
This started from a first-cut patch that reused the OpenAI backend under a new id. Run end to end against a live
llama-server(Qwen2.5-7B-Instruct on an RTX 4090), it failed in four ways, all fixed here:providers.llamacpp.mcpServers). codeoid's own memory tools stay on.Verified
READY), a turn that called the memory recall tool and answered from it, and a Claude session switched onto the local backend that recalled a codeword from before the switch. The session showed the server's real 32,768-token window after the first turn.bun test: 2949 pass, 0 fail. Web: 589 pass. Typecheck and lint clean./propsparsing, empty streams, model listing, provider ids).Not in this PR
For agents: code-level detail
src/daemon/providers/llamacpp/index.ts(new):createLlamaCppProvider,fetchServerContextWindow(/props→default_generation_settings.n_ctx, per slot),propsUrl,strictJsonSchema+llamaCppTools,allowedMcpServers.OpenAIProvidergains optionalid,displayName,modelFilter,fallbackModels,contextWindow,toolsTransform; defaults keep theopenaibackend byte-identical. Empty-stream (zero chunks) now throws. Error prefix usesdisplayName.SessionMcpToolstakesMcpServerSource = Pick<McpRegistry, "forBackend">so a filtered view can be passed.PROVIDER_DEFAULT_WINDOW.llamacpp = 4_096.providers.llamacpp { enabled=false, baseUrl, apiKey="sk-local", model?, mcpServers=[] }; settings tab added;ENABLE_KEYentry so a disabled default names its toggle.🤖 Generated with Claude Code