Skip to content

feat: run sessions on a local llama.cpp server - #365

Merged
saucam merged 2 commits into
mainfrom
feat/llamacpp-provider
Oct 10, 2026
Merged

saucam merged 2 commits into
mainfrom
feat/llamacpp-provider

Conversation

@saucam

@saucam saucam commented Oct 10, 2026

Copy link
Copy Markdown
Collaborator

What this does

Adds a llamacpp backend so a codeoid session can run on an open-weight model served on your own machine by llama.cpp's llama-server — no API key, no per-token cost, nothing leaving the machine except what your MCP servers send.

It is off by default. Turning it on is two steps, documented in docs/local-models.md:

  1. Start llama-server -m model.gguf --jinja -ngl 99 -c 32768.
  2. Set providers.llamacpp.enabled: true (and baseUrl if not on port 8080), restart the daemon, and create a session with --provider llamacpp.

It is a separate backend from openai, so a local server and a cloud OpenAI-compatible gateway can be enabled side by side.

What testing against a real server changed

This started from a first-cut patch that reused the OpenAI backend under a new id. Run end to end against a live llama-server (Qwen2.5-7B-Instruct on an RTX 4090), it failed in four ways, all fixed here:

Found live Fix
Every turn failed with HTTP 500 once any MCP server was mounted. llama.cpp rejects a tool parameter declared as "any value" without a type, which MCP servers commonly emit. Tool schemas are rewritten into an equivalent form llama.cpp accepts; tools without a description get one, because the model's chat template requires it.
Six ordinary MCP servers added ~42,000 tokens of tool definitions to every prompt — more than the model's whole 32k context. External MCP servers are opt-in per server for the local backend (providers.llamacpp.mcpServers). codeoid's own memory tools stay on.
The context window was guessed at a cloud-sized 128k, so history carried over on a switch or fork would overflow the server. The server's real per-request context is read after every turn; before the first, codeoid assumes llama-server's default of 4096.
An over-context error from older llama-server builds was silently recorded as a successful turn with no answer. A response that ends before sending anything now fails the turn and says the likely cause.

Verified

  • Live run, new code: a plain turn (READY), a turn that called the memory recall tool and answered from it, and a Claude session switched onto the local backend that recalled a codeword from before the switch. The session showed the server's real 32,768-token window after the first turn.
  • bun test: 2949 pass, 0 fail. Web: 589 pass. Typecheck and lint clean.
  • 15 new tests, each pinned to a failure seen live (untyped schemas, missing descriptions, MCP opt-in, /props parsing, empty streams, model listing, provider ids).

Not in this PR

  • The daemon never asks stateless backends (openai, gemini, llamacpp) for their model list, so the picker is empty for all three. This was already the case; local models barely need it, because the server serves the one model it loaded.
  • A session switched onto a backend with a much smaller window can auto-rotate on its first turn, because the previous backend's context size is still recorded. That behaviour already existed, but small local windows make it more likely.
For agents: code-level detail
  • src/daemon/providers/llamacpp/index.ts (new): createLlamaCppProvider, fetchServerContextWindow (/props → default_generation_settings.n_ctx, per slot), propsUrl, strictJsonSchema + llamaCppTools, allowedMcpServers.
  • OpenAIProvider gains optional id, displayName, modelFilter, fallbackModels, contextWindow, toolsTransform; defaults keep the openai backend byte-identical. Empty-stream (zero chunks) now throws. Error prefix uses displayName.
  • SessionMcpTools takes McpServerSource = Pick<McpRegistry, "forBackend"> so a filtered view can be passed.
  • PROVIDER_DEFAULT_WINDOW.llamacpp = 4_096.
  • Config: providers.llamacpp { enabled=false, baseUrl, apiKey="sk-local", model?, mcpServers=[] }; settings tab added; ENABLE_KEY entry so a disabled default names its toggle.

🤖 Generated with Claude Code

saucam and others added 2 commits October 11, 2026 02:15
A new `llamacpp` backend drives a `llama-server` the user runs, over its
OpenAI-compatible API: OpenAIProvider under its own id, so it sits beside
a cloud `openai` gateway instead of replacing it. Off by default.

Built from a first-cut patch, then run against a real llama-server
(Qwen2.5-7B-Instruct), which surfaced four things the first cut missed:

- Every turn failed with a 500 when MCP servers were mounted. llama.cpp
  compiles tool schemas into a grammar and rejects an annotation-only
  subschema like `{ "description": "any value" }`, which MCP servers emit
  for untyped parameters. strictJsonSchema spells "any value" out as every
  JSON type, the same constraint in a form it accepts, and every tool gets
  a description because Qwen-style chat templates require one.
- External MCP tools overflowed the model before the user typed: six
  imported servers came to ~42k tokens of tool definitions against a 32k
  context. External servers are now opt-in per server
  (`providers.llamacpp.mcpServers`); built-in memory tools stay on.
- The context window was a cloud-sized 128k guess. The server's real
  per-request window is read from `/props` after every turn, and the
  pre-first-turn floor is llama-server's own default of 4096, so seeds on
  a switch or fork, the context meter and auto-rotate size from what the
  server can actually hold.
- An over-context error from older llama-server builds arrives as a line
  the OpenAI SDK drops, so the turn "succeeded" with no answer. A stream
  that ends before its first chunk now fails the turn with the likely
  cause.

Also: the local model list is no longer filtered to gpt-*/o* ids (and an
unreachable server lists nothing rather than GPT models), errors are
prefixed with the backend's own name, and the session tenant is passed so
OAuth MCP servers resolve credentials.

User guide in docs/local-models.md; the CLAUDE.md section is the
maintainer summary.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@saucam
saucam merged commit 528b819 into main Oct 10, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants