diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5bc911d..c0d4fb9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -67,8 +67,8 @@ unsigned commits fail before they are pushed. - New behaviour needs a test. [docs/TESTING.md](docs/TESTING.md) records what each existing test covers - Wire-format behaviour is pinned against the published OpenAI and TypeSafe schemas; - if you change a route's shape, update the spec tests and the deliberate-exceptions - list in the README together + if you change a route's shape, update the spec tests and the README's list of + unsupported requests together ## Code of Conduct diff --git a/README.md b/README.md index 9f8bf68..019a235 100644 --- a/README.md +++ b/README.md @@ -4,13 +4,16 @@ Decision APIs for [Granite Switch](https://github.com/generative-computing/grani vLLM plugin. Ask a question about some text and get back probabilities, from the same server and GPU cache that already serves `/v1/chat/completions`. -- `POST /v1/decisions` follows [OpenAI's Decisions API](https://developers.openai.com/api/docs/guides/decisions). - The official OpenAI SDKs work once you change `base_url`. +- `POST /v1/decisions` follows [OpenAI's Decisions API](https://developers.openai.com/api/docs/guides/decisions) + for text input, and the official OpenAI SDKs work once you change `base_url`. A few parts of the API aren't + supported, and those requests get a clear error ([details](#what-v1decisions-doesnt-support)). - `POST /v1/systemone` follows [jev's system-one API](https://docs.typesafe.ai/api). Each question is answered with one engine read of the adapter's single-token option labels, so callers do not need a prompt format, an invocation marker or a label scheme. +With the public Granite Switch checkpoints, results are comparable with those on their model cards. + You can run it as a container or add it to a vLLM install you already have. Start with [Quick start](#quick-start). What has and has not been tested is in [docs/STATUS.md](docs/STATUS.md), and how to run the tests is in [docs/TESTING.md](docs/TESTING.md). @@ -109,6 +112,25 @@ curl http://localhost:8000/v1/decisions -H "Content-Type: application/json" -d ' equivalent `/v1/systemone` request. For equivalent questions both routes build the same prompt and return the same probabilities. +### More request options + +`/v1/systemone` also accepts the extra fields on vLLM's own endpoint, so most clients written for vLLM work as they +are. + +- **`cache_salt`** keeps a request's prefix cache separate from everyone else's. The cache is shared, so a fast + response (or a `cached_tokens` count) can show that someone has already sent the same prompt. If you serve several + tenants, have your gateway set a long random `cache_salt` for each. It's a privacy measure, not authentication. + `/v1/decisions` accepts it too. +- **`chat_template_kwargs`** may only be `{"enable_thinking": false}`. The server always chooses the adapter. +- **`joint`, `images`, `audio`, `videos`, `priority`, `request_id`** are accepted at their defaults. This model reads + text, one question at a time, so other values get a 422. +- **`model`** is required here, though vLLM makes it optional. + +Each question in a `/v1/systemone` response also carries `diagnostics`: `label_mass`, how much probability the model +put on the option labels (a low value usually means a wrong marker or layout), and `argmax_is_label`. OpenAI's +response schema doesn't allow extra fields, so `/v1/decisions` has none. `confidence` uses jev's formula, so it isn't +comparable with vLLM's. + ## Secure it before you expose it The server listens on `0.0.0.0` and, as shipped, asks for no credentials on any route. Put it behind a @@ -219,10 +241,11 @@ same prompt, served from the prefix cache. - **Caller text is escaped.** Special tokens and the invocation marker inside the state, instructions or descriptions are broken with an invisible U+2060, so they cannot forge turns or move the adapter's activation point. -- **Input.** `input` messages must have the role `user`, and images are rejected, because the trained - layout has one text state turn. `/v1/systemone` rejects unknown fields and `box` questions. +- **Input.** `input` is a string or a list of user messages. A message can hold text and `input_json` parts, which + are written as compact JSON, as an object `state` is on `/v1/systemone`. A question's `instructions` can be a + string or text and JSON parts. `/v1/systemone` rejects unknown fields and `box` questions. - **Size.** The state, instructions and options of each question are counted before any engine work: - at most 200 questions, 32k tokens per question and 64k per request (262k with + at most 400 questions, 32k tokens per question and 64k per request (262k with `TypeSafe-Danger-Zone-Long-Context: true` on `/v1/systemone`), all capped by `--max-model-len`. Bodies over 32 MiB, nesting deeper than 64 and non-finite numbers are rejected. - **Usage.** A `choice` with more than 128 options runs as two engine requests over the same prompt, @@ -235,9 +258,22 @@ same prompt, served from the prefix cache. - **Cancellation.** A client that disconnects cancels its request's engine work. - **Errors.** `/v1/decisions` uses OpenAI's error shape and `/v1/systemone` uses jev's `HTTPValidationError` for 422s. Engine errors keep vLLM's status code. +- **Repeat requests.** The same request can return slightly different probabilities, depending on how it is + batched and whether it hits the prefix cache, so a near-tie can flip. A different GPU or tensor-parallel size can + differ a little more. vLLM's `VLLM_BATCH_INVARIANT=1` aims to make repeats exact; we haven't tried it here. - **Not provided.** Authentication (use vLLM's `--api-key`), rate limiting and `refusal` answers. - Engine reads are not bit-identical between batched and solo requests, so repeated calls can differ in - the third decimal place. + +### What /v1/decisions doesn't support + +The decision adapter reads one block of text and a question, and this server keeps to that layout so results stay +predictable. So a few requests that OpenAI's schema allows get a 400 that names the part: + +- developer, system and assistant messages (put what the questions are about in a user message) +- tool calls and tool outputs +- images, in `input` or in a question's `instructions` +- choices that would merge into one: two with the same value, or `"true"` and `true` + +The route never returns `refusal` answers, and `/v1/systemone` likewise refuses request-level `instructions`. ## Upstream status diff --git a/docs/STATUS.md b/docs/STATUS.md index ee1e27f..f8bf24f 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -15,14 +15,14 @@ and spec suites and the 8b battery (`e2e_check`, `probe_check`, `accuracy_check` | Area | How | Result | |---|---|---| | Unit and route tests | `pytest`, engine stubbed, in CI | pass | -| `/v1/decisions` against OpenAI's OpenAPI schema | `tests/test_openai_spec.py` (41 cases), plus the same request set live | pass, with the exceptions listed in the README | +| `/v1/decisions` against OpenAI's OpenAPI schema | `tests/test_openai_spec.py` (against the general-availability schema), plus the same request set live | pass, with the exceptions listed in the README | | `/v1/systemone` against jev's wire format | `tests/test_systemone_spec.py` (24 cases) against `typesafe-sdk`'s generated models, stubbed | pass; the plugin also rejects unknown fields, which jev's format allows | -| Live wire battery | `scripts/e2e_check.py` (11 checks) | 11/11 | +| Live wire battery | `scripts/e2e_check.py` (12 checks, including vLLM's request fields and `cache_salt`) | 12/12 | | Prompt rendering | server prompt token counts against an offline render with the checkpoint's tokenizer and chat template | identical | | Prefix-cache sharing | `tests/test_prefix_sharing.py`, and `cached_tokens` live, cross-checked with `/metrics` | as documented in the README | | Route parity | 800 labelled items through `/v1/decisions` and through the same prompt over chat completions | no item answers differently | | Escaping, size guard, disconnect cancellation, error shapes | live probes, now `scripts/probe_check.py` | as documented in the README | -| Question cap and `DECISION_MAX_CONCURRENT_REQUESTS` | live: 200 questions accepted, 201 rejected; a 32-request burst against a cap of 8 gave 8 accepted and 24 rejected with 503 | pass | +| Question cap and `DECISION_MAX_CONCURRENT_REQUESTS` | live: 400 questions accepted, 401 rejected; a 32-request burst against a cap of 8 gave 8 accepted and 24 rejected with 503 | pass | | `DECISION_EXPERIMENTAL_SYSTEM_PROMPT=1` | live: the 800-item comparison and `e2e_check.py --system-prompt` | answers identical to the same prompt over chat completions; decisions about one state still share the cache, chat no longer does | | FlashAttention 3 | FA2 and FA3 servers side by side: the 800-item comparison, 40 cached one-token reads (the case in granite-switch#139), load at 8 and 32 clients | no difference beyond batching noise in answers or speed; FA2 kept in `serve.sh` while #139 is open | | Load | 1 to 128 concurrent clients, three-question requests | about 140 requests/s plateau, no errors | @@ -72,6 +72,9 @@ show that the serving path is identical to the one that produced the card's figu ## Known gaps +- **`/v1/decisions` covers OpenAI's schema for text only.** Images, developer, system and assistant + messages and tool calls get a 400, and `/v1/systemone` refuses request-level `instructions`. The adapter + reads one state turn and a question, and nothing else is known to work. The README lists each case. - **Speculative decoding is untested and may crash the engine.** Decision reads use `logprob_token_ids`; vLLM issue #42592 (fix #44727, open) crashes on that whenever a speculative-decoding request shares the batch. We have not run MTP, so whether the decision diff --git a/docs/TESTING.md b/docs/TESTING.md index c11bad0..5ab0d54 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -35,10 +35,12 @@ it needs `gitleaks` on `PATH` for the last of those. question) and error shapes. - `tests/test_openai_spec.py`: `/v1/decisions` requests and responses validated with a JSON Schema validator against OpenAI's published schema. The schema is downloaded on - first run from `openai/openai-openapi` (MIT) at a pinned commit, checked against a - pinned SHA-256 and cached in `~/.cache/granite-switch-decision`; it is never committed. + first run from `openai/openai-openapi` (MIT) at a pinned commit (the general-availability + revision), checked against a pinned SHA-256 and cached in `~/.cache/granite-switch-decision`; + it is never committed. To move the pin, set `OPENAI_COMMIT` in `tests/specs.py` and compute + `OPENAI_SHA256` with `curl -fsSL https://raw.githubusercontent.com/openai/openai-openapi//openapi.json | shasum -a 256`. Offline, the test skips and says how to supply a local copy (`DECISION_OPENAI_SPEC`). - Deliberate limits are listed as named exceptions. + Unsupported requests are listed as named cases, matching the README's "What /v1/decisions doesn't support". - `tests/test_systemone_spec.py`: `/v1/systemone` validated against jev's wire models from TypeSafe's MIT-licensed `typesafe-sdk` (a dev dependency), which TypeSafe generates from its API spec. Those models omit the spec's bounds on choice options (1-255) and @@ -68,7 +70,7 @@ Python standard library unless noted. | Command | What it checks | |---|---| -| `python3 scripts/e2e_check.py --base-url URL` | Both routes' answer shapes and parity, input flattening, boolean values, error shapes, usage, chat on the same server, prefix-cache hits. Add `--system-prompt` when the server runs with `DECISION_EXPERIMENTAL_SYSTEM_PROMPT=1`. | +| `python3 scripts/e2e_check.py --base-url URL` | Both routes' answer shapes and parity, input flattening, error shapes, usage, vLLM's request fields and `cache_salt`, chat on the same server, prefix-cache hits. Add `--system-prompt` when the server runs with `DECISION_EXPERIMENTAL_SYSTEM_PROMPT=1`. | | `python3 scripts/probe_check.py --base-url URL` | Question cap, size guard (no engine work for oversized requests), body limits, special-token and marker escaping, options without descriptions, disconnect cancellation. | | `python3 scripts/probe_check.py --base-url URL --max-concurrent N` | Adds the concurrency limit: a burst against `DECISION_MAX_CONCURRENT_REQUESTS=N` gets 200s and 503s. | | `python3 scripts/probe_check.py --base-url URL --load 30` | Adds a load test at 1, 8 and 32 clients with chat latency alongside. Reported, not asserted. | diff --git a/scripts/e2e_check.py b/scripts/e2e_check.py index aec0fc7..e670e3b 100755 --- a/scripts/e2e_check.py +++ b/scripts/e2e_check.py @@ -19,12 +19,19 @@ parity the same questions through both routes agree: same choice pick, same per-option/level probabilities, same predicate probability, same score - input_flattening a message-list input answers identically to the string + input_flattening a message-list input, an input_json part and instruction parts + answer identically to their plain-string equivalents boolean_values boolean choice values round-trip as booleans usage usage is present and non-zero errors 400s with OpenAI's error shape (empty questions, image input, 11 score levels) and jev's 422 shape on the systemone route + vllm_fields vLLM's request fields on /v1/systemone: enable_thinking false + accepted; request-level instructions, adapter_name, other + template kwargs, a bad cache_salt, joint and images refused; + system messages refused on /v1/decisions; a + cache_salt gives its own prefix cache (needs + --enable-prompt-tokens-details) chat /v1/chat/completions still works on the shared server (skip with --skip-chat) prefix_cache prefix-cache hits for a decision after a chat or a @@ -198,6 +205,12 @@ def check_systemone_shapes(client: Client, model: str) -> Check: check.expect_equal(severity.get("type"), "score", "severity type") check.expect(isinstance(severity.get("score"), (int, float)), "severity score") check.expect_equal(len(severity.get("probabilities", {})), 3, "severity probabilities") + diagnostics = body.get("diagnostics", {}) + check.expect(set(diagnostics) == set(answers), "diagnostics keys match the answers") + for name, entry in diagnostics.items(): + mass = entry.get("label_mass") + check.expect(isinstance(mass, (int, float)) and 0.0 <= mass <= 1.0 + 1e-6, f"{name} label_mass in [0, 1]") + check.expect(isinstance(entry.get("argmax_is_label"), bool), f"{name} argmax_is_label is a boolean") return check @@ -269,6 +282,40 @@ def check_input_flattening(client: Client, model: str, tol: float) -> Check: "string vs message-list probability", tol, ) + + # A JSON part is the compact JSON text of an object state, and instruction parts join like input parts. + obj = {"ticket": "My payouts have failed three times.", "attempts": 3} + _, text_form = client.post( + "/v1/decisions", {"model": model, "input": json.dumps(obj, ensure_ascii=False), "questions": [question]} + ) + _, json_form = client.post( + "/v1/decisions", + { + "model": model, + "input": [{"role": "user", "content": [{"type": "input_json", "data": obj}]}], + "questions": [question], + }, + ) + check.expect_close( + text_form["answers"][0]["probability"], + json_form["answers"][0]["probability"], + "JSON text vs input_json probability", + tol, + ) + parts = [{"type": "input_text", "text": "Is this about"}, {"type": "input_text", "text": "payouts?"}] + _, plain = client.post( + "/v1/decisions", + {"model": model, "input": STATE, "questions": [{**question, "instructions": "Is this about\n\npayouts?"}]}, + ) + _, split = client.post( + "/v1/decisions", {"model": model, "input": STATE, "questions": [{**question, "instructions": parts}]} + ) + check.expect_close( + plain["answers"][0]["probability"], + split["answers"][0]["probability"], + "string vs instruction-parts probability", + tol, + ) return check @@ -351,6 +398,57 @@ def check_errors(client: Client, model: str) -> Check: return check +def check_vllm_fields(client: Client, model: str) -> Check: + """vLLM's own request fields: accepted where the plugin supports them, refused (and never + forwarded) where it cannot, and `cache_salt` really partitions the prefix cache.""" + check = Check("vllm_fields") + body = {**SYSTEMONE_BODY, "model": model} + status, _ = client.post("/v1/systemone", {**body, "chat_template_kwargs": {"enable_thinking": False}}) + check.expect_equal(status, 200, "enable_thinking: false") + status, _ = client.post("/v1/systemone", {**body, "instructions": "Answer strictly."}) + check.expect_equal(status, 422, "request-level instructions") + status, _ = client.post("/v1/systemone", {**body, "chat_template_kwargs": {"adapter_name": "guardian-core"}}) + check.expect_equal(status, 422, "adapter_name in chat_template_kwargs") + status, _ = client.post("/v1/systemone", {**body, "chat_template_kwargs": {"documents": [{"text": "x"}]}}) + check.expect_equal(status, 422, "unsupported chat_template_kwargs key") + status, _ = client.post("/v1/systemone", {**body, "cache_salt": "a/b"}) + check.expect_equal(status, 422, "cache_salt with a forbidden character") + status, _ = client.post("/v1/systemone", {**body, "joint": True}) + check.expect_equal(status, 422, "joint: true") + status, _ = client.post("/v1/systemone", {**body, "images": ["data:image/png;base64,AA"]}) + check.expect_equal(status, 422, "images") + status, _ = client.post("/v1/decisions", {**DECISIONS_BODY, "model": model, "instructions": "x"}) + check.expect_equal(status, 400, "instructions on /v1/decisions") + system = [{"role": "system", "content": "Be terse."}, {"role": "user", "content": STATE}] + status, _ = client.post("/v1/decisions", {**DECISIONS_BODY, "model": model, "input": system}) + check.expect_equal(status, 400, "system message on /v1/decisions") + + # A fresh salt starts with an empty cache, the same salt then hits it, and another salt (or none) does not. + text = f"Ticket {uuid.uuid4().hex}. " + TICKET * 6 + salted = { + "model": model, + "input": text, + "cache_salt": f"salt-{uuid.uuid4().hex}", + "questions": [{"type": "predicate", "instructions": "Does this need urgent attention?"}], + } + + def cached(request: dict) -> int: + _, response = client.post("/v1/decisions", request) + return response.get("usage", {}).get("input_tokens_details", {}).get("cached_tokens", 0) + + first, second = cached(salted), cached(salted) + other = cached({**salted, "cache_salt": f"salt-{uuid.uuid4().hex}"}) + unsalted = cached({k: v for k, v in salted.items() if k != "cache_salt"}) + check.notes.append( + f"cache_salt: first {first}, same salt again {second}, other salt {other}, no salt {unsalted} cached tokens" + ) + check.expect(first == 0, f"a fresh salt read {first} tokens from the cache") + check.expect(second > 0, "the same salt did not hit the cache") + check.expect(other == 0, f"another salt read {other} tokens from the cache") + check.expect(unsalted == 0, f"no salt read {unsalted} tokens from the salted cache") + return check + + def check_chat(client: Client, model: str) -> Check: check = Check("chat") status, body = client.post( @@ -535,6 +633,7 @@ def main() -> int: check_boolean_values(client, model), check_usage(client, model), check_errors(client, model), + check_vllm_fields(client, model), ] if not args.skip_chat: checks.append(check_chat(client, model)) diff --git a/scripts/probe_check.py b/scripts/probe_check.py index d2ecb13..46fa240 100644 --- a/scripts/probe_check.py +++ b/scripts/probe_check.py @@ -9,7 +9,7 @@ Exit code 0 means every check passed. Checks: - question_cap 200 questions accepted on /v1/systemone, 201 rejected (422) + question_cap 400 questions accepted on /v1/systemone, 401 rejected (422) size_guard a short state with long options is rejected before engine work body_limits oversized body, deep nesting, Content-Encoding, lone surrogates escaping special tokens and the invocation marker in caller text are read @@ -89,13 +89,13 @@ def predicate(text: str) -> dict: def check_question_cap(p: Probe) -> Result: r = Result("question_cap") - questions = {f"q{i}": {"type": "noul", "instructions": f"Question {i}?"} for i in range(201)} + questions = {f"q{i}": {"type": "noul", "instructions": f"Question {i}?"} for i in range(401)} status, _ = p.post( - "/v1/systemone", {"model": p.model, "state": "x", "questions": dict(list(questions.items())[:200])} + "/v1/systemone", {"model": p.model, "state": "x", "questions": dict(list(questions.items())[:400])} ) - r.expect(status == 200, f"200 questions: status {status}, want 200") + r.expect(status == 200, f"400 questions: status {status}, want 200") status, _ = p.post("/v1/systemone", {"model": p.model, "state": "x", "questions": questions}) - r.expect(status == 422, f"201 questions: status {status}, want 422") + r.expect(status == 422, f"401 questions: status {status}, want 422") return r diff --git a/src/granite_switch_decision/plugin.py b/src/granite_switch_decision/plugin.py index 2b56b48..d781226 100644 --- a/src/granite_switch_decision/plugin.py +++ b/src/granite_switch_decision/plugin.py @@ -18,11 +18,11 @@ import math import os from http import HTTPStatus -from typing import TYPE_CHECKING, Any, Literal, NamedTuple +from typing import TYPE_CHECKING, Annotated, Any, Literal, NamedTuple from fastapi import APIRouter, FastAPI, Request from fastapi.responses import JSONResponse, Response -from pydantic import BaseModel, ConfigDict, model_validator +from pydantic import BaseModel, ConfigDict, Field, StrictBool, field_validator, model_validator from pydantic import ValidationError as PydanticValidationError from starlette.routing import Match @@ -51,7 +51,7 @@ # vLLM's MAX_LOGPROB_TOKEN_IDS; a choice with more labels is read in chunks of this size. MAX_LABEL_OPTIONS = 128 # Questions per request on both routes (OpenAI's limit on /v1/decisions). -MAX_QUESTIONS = 200 +MAX_QUESTIONS = 400 _MAX_CONCURRENT_ENV = "DECISION_MAX_CONCURRENT_REQUESTS" # Engine reads in flight per request, so one wide request cannot flood the engine. MAX_READS_PER_REQUEST = 16 @@ -132,13 +132,75 @@ def _check_criteria(self) -> Question: return self +_CACHE_SALT_FORBIDDEN = frozenset("@/\\\x00") +_MAX_CACHE_SALT = 128 + + +def _check_cache_salt(value: str | None) -> str | None: + """vLLM's own rule for a cache salt, checked up front so a bad value is a 4xx, not a 500.""" + if value is None: + return None + if not value or len(value) > _MAX_CACHE_SALT or any(c in _CACHE_SALT_FORBIDDEN for c in value): + raise ValueError( + "`cache_salt` must be a non-empty string of at most 128 characters without '@', '/', '\\' or NUL" + ) + return value + + +def _check_joint(value: bool) -> bool: + if value: + raise ValueError("`joint` is not supported: this model reads each question on its own") + return value + + +_NO_REQUEST_CONTEXT = ( + "request-level `instructions` and developer or system messages are not supported: the decision adapter reads " + "the state and each question's own instructions. Put the context in the state." +) + + class SystemOneRequest(BaseModel): + """jev's request plus the fields vLLM's `/v1/systemone` takes. `chat_template_kwargs` is checked and + never forwarded, because the chat template renders some of its keys unescaped. Request-level + `instructions` and non-empty media are refused.""" + model_config = ConfigDict(extra="forbid") state: Any model: str questions: dict[str, Question] + instructions: Any = None + chat_template_kwargs: dict[str, Any] | None = None + cache_salt: str | None = None + joint: StrictBool = False + images: list[Any] | None = None + audio: list[Any] | None = None + videos: list[Any] | None = None + priority: int = 0 + request_id: str | None = None + + @model_validator(mode="after") + def _check_vllm_fields(self) -> SystemOneRequest: + if self.instructions is not None: + raise ValueError(_NO_REQUEST_CONTEXT) + _check_cache_salt(self.cache_salt) + _check_joint(self.joint) + for field in ("images", "audio", "videos"): + if getattr(self, field): + raise ValueError(f"`{field}` is not supported: this server serves a text decision adapter") + if self.priority != 0: + raise ValueError("`priority` is not supported: requests are not scheduled by priority") + kwargs = self.chat_template_kwargs or {} + if "adapter_name" in kwargs: + raise ValueError("`chat_template_kwargs.adapter_name` is set by the server") + unknown = sorted(set(kwargs) - {"enable_thinking"}) + if unknown: + raise ValueError(f"`chat_template_kwargs` keys {unknown} are not supported: only `enable_thinking: false`") + if kwargs.get("enable_thinking"): + raise ValueError("`chat_template_kwargs.enable_thinking` must be false: a read is the reply's first token") + return self + @model_validator(mode="after") def _question_count(self) -> SystemOneRequest: # jev requires at least one question and sets no maximum; each is an engine request. @@ -169,16 +231,64 @@ class DecisionLevel(BaseModel): description: str | None = None +class DecisionCacheBreakpoint(BaseModel): + model_config = ConfigDict(extra="forbid") + + mode: Literal["explicit"] = "explicit" + + +class DecisionQuestionText(BaseModel): + model_config = ConfigDict(extra="forbid") + + type: Literal["input_text"] = "input_text" + text: str + + +class DecisionQuestionImage(BaseModel): + model_config = ConfigDict(extra="forbid") + + type: Literal["input_image"] = "input_image" + image_url: str + detail: Literal["low", "high", "auto", "original"] | None = None + + +class DecisionInputJson(BaseModel): + """A JSON object, or an array of objects, rendered as compact JSON text like an object `state`.""" + + model_config = ConfigDict(extra="forbid") + + type: Literal["input_json"] = "input_json" + data: dict[str, Any] | list[dict[str, Any]] + + +class DecisionInputText(DecisionQuestionText): + """`prompt_cache_breakpoint` is accepted and ignored: vLLM's prefix cache reuses + every full block of a matching prefix on its own, with no breakpoints to place.""" + + prompt_cache_breakpoint: DecisionCacheBreakpoint | None = None + + +class DecisionInputImage(DecisionQuestionImage): + prompt_cache_breakpoint: DecisionCacheBreakpoint | None = None + + +_QuestionPart = Annotated[ + DecisionQuestionText | DecisionQuestionImage | DecisionInputJson, Field(discriminator="type") +] +_MessagePart = Annotated[DecisionInputText | DecisionInputImage | DecisionInputJson, Field(discriminator="type")] + + class DecisionQuestion(BaseModel): """One OpenAI Decisions question (`predicate`/`choice`/`score`), with the field - constraints of OpenAI's published spec: `instructions` required, 2-255 unique - choices, at most 10 score levels. `name` is optional, since answers are positional, so - duplicate or missing names cannot collide (the response echoes the name or null).""" + constraints of OpenAI's published spec: `instructions` required (a string, or text and + JSON parts), 2-255 unique choices, at most 10 score levels. `name` is optional, since + answers are positional, so duplicate or missing names cannot collide (the response + echoes the name or null).""" model_config = ConfigDict(extra="forbid") type: Literal["predicate", "choice", "score"] - instructions: str + instructions: str | Annotated[list[_QuestionPart], Field(min_length=1)] name: str | None = None choices: list[DecisionChoice] | None = None levels: list[DecisionLevel] | None = None @@ -208,30 +318,15 @@ def _check_type_fields(self) -> DecisionQuestion: return self -class DecisionCacheBreakpoint(BaseModel): - model_config = ConfigDict(extra="forbid") - - mode: Literal["explicit"] = "explicit" - - -class DecisionInputPart(BaseModel): - """`prompt_cache_breakpoint` is accepted and ignored: vLLM's prefix cache reuses - every full block of a matching prefix on its own, with no breakpoints to place.""" - - model_config = ConfigDict(extra="forbid") - - type: Literal["input_text", "input_image"] - text: str | None = None - image_url: str | None = None - prompt_cache_breakpoint: DecisionCacheBreakpoint | None = None - - class DecisionInputMessage(BaseModel): model_config = ConfigDict(extra="forbid") type: Literal["message"] = "message" role: Literal["user"] - content: str | list[DecisionInputPart] + content: str | list[_MessagePart] + + +_UNSUPPORTED_ITEM_TYPES = ("function_call", "function_call_output", "custom_tool_call", "custom_tool_call_output") class DecisionRequest(BaseModel): @@ -249,9 +344,36 @@ class DecisionRequest(BaseModel): prompt_cache_options: Any = None safety_identifier: str | None = None + # vLLM's extensions to this route (its own `/v1/decisions` accepts exactly these two). + cache_salt: str | None = None + joint: StrictBool = False + + @field_validator("input", mode="before") + @classmethod + def _name_the_unsupported_items(cls, value: Any) -> Any: + """Valid in OpenAI's schema but not read here: say which part, and what to do instead.""" + if isinstance(value, list): + for item in value: + if not isinstance(item, dict): + continue + kind = item["type"] if item.get("type") in _UNSUPPORTED_ITEM_TYPES else item.get("role") + if kind in _UNSUPPORTED_ITEM_TYPES or kind in ("developer", "system", "assistant"): + what = f"`{kind}` items" if kind in _UNSUPPORTED_ITEM_TYPES else f"`{kind}` messages" + raise ValueError( + f"{what} are not supported: the decision adapter reads user text. " + "Put what the questions are about in a user message." + ) + return value + + @model_validator(mode="after") + def _check_vllm_extensions(self) -> DecisionRequest: + _check_cache_salt(self.cache_salt) + _check_joint(self.joint) + return self + @model_validator(mode="after") def _check_questions(self) -> DecisionRequest: - # OpenAI: "Ask 1-200 questions about the input. ... Answers come back in the + # OpenAI: "Ask 1-400 questions about the input. ... Answers come back in the # same order." if not 1 <= len(self.questions) <= MAX_QUESTIONS: raise ValueError(f"`questions` must contain 1-{MAX_QUESTIONS} entries") @@ -566,6 +688,14 @@ def _error_response_type() -> type: return ErrorResponse +def _sampled_token_id(resp) -> int | None: + """The id of the token the engine sampled (`token_id:`), or None if it is not reported.""" + try: + return int(resp.choices[0].logprobs.content[0].token.split(":", 1)[1]) + except (AttributeError, IndexError, TypeError, ValueError): + return None + + def _label_logprobs_by_id(top_logprobs) -> dict[int, float]: """`token_id:` entries from a response's top_logprobs, keyed by id.""" try: @@ -577,15 +707,30 @@ def _label_logprobs_by_id(top_logprobs) -> dict[int, float]: _LABEL_MASS_WARN = 0.9 -def _log_label_mass(name: str, label_logprobs: list[float]) -> None: - """Log the probability mass the option labels hold before normalising. Low mass means the - model wanted a token outside the labels (a wrong marker, a broken layout). Observation only: - answers are never gated on it.""" +def _log_label_mass(name: str, label_logprobs: list[float]) -> float: + """Log, and return, the probability mass the option labels hold before normalising. Low mass + means the model wanted a token outside the labels (a wrong marker, a broken layout). + Observation only: answers are never gated on it.""" mass = sum(math.exp(lp) for lp in label_logprobs) if mass < _LABEL_MASS_WARN: _log.warning("decision %r: option labels hold only %.3f of the probability mass", name, mass) else: _log.debug("decision %r: label mass %.4f", name, mass) + return mass + + +class QuestionDiagnostics(NamedTuple): + """vLLM's `diagnostics` for one question: the probability the model put on the option labels + over its whole vocabulary, and whether its most likely token was one of them.""" + + label_mass: float + argmax_is_label: bool + + +class ReadOptions(NamedTuple): + """Per-request settings every engine read of a request shares.""" + + cache_salt: str | None = None def _check_activation(tok, renderer, adapter_name: str) -> None: @@ -632,6 +777,7 @@ async def _fetch_label_logprobs( adapter_name: str, msgs: list[dict[str, str]], label_ids_chunk: list[int], + cache_salt: str | None = None, ): """One `create_chat_completion` call for the logprobs of `label_ids_chunk` over the rendered `msgs`. Returns the raw response; a question with more than @@ -651,7 +797,9 @@ async def _fetch_label_logprobs( top_logprobs=1, logprob_token_ids=label_ids_chunk, return_tokens_as_token_ids=True, + # Never caller-supplied: the template renders other keys unescaped, and the adapter is the server's. chat_template_kwargs={"adapter_name": adapter_name}, + cache_salt=cache_salt, ) resp = await chat.create_chat_completion(req) if isinstance(resp, ErrorResponse): @@ -671,9 +819,10 @@ async def _answer_question( adapter_name: str, renderer, reads: asyncio.Semaphore, -) -> tuple[str, dict[str, Any], TokenUsage]: - """Returns (name, jev-shaped answer dict, this question's TokenUsage). `reads` bounds - the engine reads in flight across one request's questions.""" + options: ReadOptions = ReadOptions(), +) -> tuple[str, dict[str, Any], TokenUsage, QuestionDiagnostics]: + """Returns (name, jev-shaped answer dict, this question's TokenUsage, its diagnostics). + `reads` bounds the engine reads in flight across one request's questions.""" if q.type == "choice": keys = list(q.criteria) # An option without a description is described by its key, as vLLM's native strategy does. @@ -691,7 +840,7 @@ async def _answer_question( # Over 128 labels: one request per chunk, same prompt, so later chunks hit the prefix cache. async def read(chunk: list[int]): async with reads: - return await _fetch_label_logprobs(chat, served_model, adapter_name, msgs, chunk) + return await _fetch_label_logprobs(chat, served_model, adapter_name, msgs, chunk, options.cache_salt) tasks = [asyncio.ensure_future(read(chunk)) for chunk in _chunk_label_ids(label_ids)] try: @@ -713,8 +862,9 @@ async def read(chunk: list[int]): # Every chunk is a separate engine request over the same prompt, so usage counts each. usage = _sum_usage([_chunk_usage(resp.usage) for resp in responses]) - _log_label_mass(name, [by_id[tid] for tid in label_ids]) + label_mass = _log_label_mass(name, [by_id[tid] for tid in label_ids]) probs = _softmax([by_id[tid] for tid in label_ids]) + diagnostics = QuestionDiagnostics(label_mass, _sampled_token_id(responses[0]) in set(label_ids)) if q.type == "choice": j = max(range(len(keys)), key=lambda i: probs[i]) @@ -735,7 +885,7 @@ async def read(chunk: list[int]): "confidence": _score_confidence(probs), } - return name, answer, usage + return name, answer, usage, diagnostics class _Limiter: @@ -786,7 +936,9 @@ async def disconnected() -> None: await _cancel_pending([work_task, watch_task]) -async def _answer_all(state, state_value: Any, questions: list[tuple[str, Question]]) -> list: +async def _answer_all( + state, state_value: Any, questions: list[tuple[str, Question]], options: ReadOptions = ReadOptions() +) -> list: """Answer every question concurrently, at most MAX_READS_PER_REQUEST engine reads at a time; if one fails, cancel the rest and re-raise.""" reads = asyncio.Semaphore(MAX_READS_PER_REQUEST) @@ -802,6 +954,7 @@ async def _answer_all(state, state_value: Any, questions: list[tuple[str, Questi state.decision_adapter_name, state.decision_renderer, reads, + options, ) ) for key, q in questions @@ -821,40 +974,45 @@ async def _cancel_pending(tasks: list[asyncio.Task]) -> None: await asyncio.gather(*tasks, return_exceptions=True) +def _flatten_parts(parts: list[Any]) -> str: + """Content parts as one string, joined with blank lines: text as it is, JSON as compact + JSON (the rule an object `state` follows on the systemone route). Image parts are + rejected, as `box` questions are on the systemone route.""" + texts: list[str] = [] + for part in parts: + if isinstance(part, (DecisionQuestionImage, DecisionInputImage)): + raise ValueError("image input is not supported: this server serves a text decision adapter") + texts.append(part.text if isinstance(part, DecisionQuestionText) else _text_of(part.data)) + return "\n\n".join(texts) + + def _decisions_input_to_state(data: str | list[DecisionInputMessage]) -> str: - """Flatten OpenAI's `input` (a string, or user messages with text/image parts) into the - state string both routes share: message texts joined with blank lines. Image parts - are rejected, as `box` questions are on the systemone route.""" + """Flatten OpenAI's `input` (a string, or user messages with text and JSON parts) into the state + string both routes share: the messages' texts joined with blank lines.""" if isinstance(data, str): return data - texts: list[str] = [] - for message in data: - if isinstance(message.content, str): - texts.append(message.content) - continue - for part in message.content: - if part.type == "input_image": - raise ValueError("image input is not supported: this server serves a text decision adapter") - if part.text is None: - raise ValueError("an `input_text` part needs a `text` string") - texts.append(part.text) - return "\n\n".join(texts) + return "\n\n".join(m.content if isinstance(m.content, str) else _flatten_parts(m.content) for m in data) + + +def _instructions_text(instructions: str | list[Any]) -> str: + return instructions if isinstance(instructions, str) else _flatten_parts(instructions) def _decisions_question_to_internal(question: DecisionQuestion, index: int) -> tuple[str, Question, dict[str, Any]]: """An OpenAI question as the internal jev question, keyed by position, plus what maps the answer back.""" + instructions = _instructions_text(question.instructions) if question.type == "predicate": - internal = Question(type="noul", instructions=question.instructions) + internal = Question(type="noul", instructions=instructions) extras: dict[str, Any] = {} elif question.type == "choice": assert question.choices is not None # the validator guarantees it for choice criteria = {_choice_key(c.value): c.description for c in question.choices} - internal = Question(type="choice", instructions=question.instructions, criteria=criteria) + internal = Question(type="choice", instructions=instructions, criteria=criteria) extras = {"value_map": {_choice_key(c.value): c.value for c in question.choices}} else: assert question.levels is not None # the validator guarantees it for score descriptions = [level.description or level.label for level in question.levels] - internal = Question(type="score", instructions=question.instructions, criteria=descriptions) + internal = Question(type="score", instructions=instructions, criteria=descriptions) extras = {"labels": [level.label for level in question.levels]} return str(index), internal, extras @@ -939,21 +1097,25 @@ async def _handle_systemone(raw_request: Request) -> Response: return _error(HTTPStatus.SERVICE_UNAVAILABLE, _BUSY) try: results = await _run_until_disconnect( - raw_request, _answer_all(state, req.state, list(req.questions.items())) + raw_request, + _answer_all(state, req.state, list(req.questions.items()), ReadOptions(req.cache_salt)), ) except EngineError as e: return _error(e.status, str(e)) if results is None: return Response(status_code=499) - answers = {name: answer for name, answer, _ in results} - usage = _sum_usage([question_usage for _, _, question_usage in results]) + answers = {name: answer for name, answer, _, _ in results} + usage = _sum_usage([question_usage for _, _, question_usage, _ in results]) return JSONResponse( content={ "model": state.decision_served_model, "answers": answers, "usage": {"input_tokens": usage.input_tokens, "output_tokens": usage.output_tokens}, + "diagnostics": { + name: {"label_mass": d.label_mass, "argmax_is_label": d.argmax_is_label} for name, _, _, d in results + }, } ) @@ -989,11 +1151,10 @@ async def _handle_decisions(raw_request: Request) -> Response: try: state_text = _decisions_input_to_state(req.input) + translated = [_decisions_question_to_internal(question, index) for index, question in enumerate(req.questions)] except ValueError as e: return _openai_error(HTTPStatus.BAD_REQUEST, str(e)) - translated = [_decisions_question_to_internal(question, index) for index, question in enumerate(req.questions)] - try: _enforce_context_limits( state.decision_tokenizer, @@ -1005,7 +1166,9 @@ async def _handle_decisions(raw_request: Request) -> Response: except ValueError as e: return _openai_error(HTTPStatus.BAD_REQUEST, str(e)) - work = _answer_all(state, state_text, [(key, internal) for key, internal, _ in translated]) + work = _answer_all( + state, state_text, [(key, internal) for key, internal, _ in translated], ReadOptions(req.cache_salt) + ) async with state.decision_limiter.slot() as admitted: if not admitted: work.close() @@ -1019,9 +1182,9 @@ async def _handle_decisions(raw_request: Request) -> Response: answers = [ _internal_answer_to_decisions(req.questions[int(key)], extras, answer) - for (_, _, extras), (key, answer, _) in zip(translated, results) + for (_, _, extras), (key, answer, _, _) in zip(translated, results) ] - usage = _sum_usage([question_usage for _, _, question_usage in results]) + usage = _sum_usage([question_usage for _, _, question_usage, _ in results]) return JSONResponse( content={ diff --git a/tests/specs.py b/tests/specs.py index 3d6cce0..d7c40f3 100644 --- a/tests/specs.py +++ b/tests/specs.py @@ -19,9 +19,9 @@ import urllib.request from pathlib import Path -OPENAI_COMMIT = "33248a638234a53846ed3ab4b2a7c77069bc1030" +OPENAI_COMMIT = "8243cc8fa6f04430fb1f1a6a2cb926fbeb4f7f26" OPENAI_URL = f"https://raw.githubusercontent.com/openai/openai-openapi/{OPENAI_COMMIT}/openapi.json" -OPENAI_SHA256 = "c09eea21c178f883549c690752aa082a9fef494a719b7a08ca85341bd033b221" +OPENAI_SHA256 = "69448733efdb426a3d1914a5cd18995d2849db2c41e20b9b14c7eb6fa84540b6" _CACHE = Path(os.environ.get("XDG_CACHE_HOME", Path.home() / ".cache")) / "granite-switch-decision" OPENAI_REASON = ( diff --git a/tests/test_decisions.py b/tests/test_decisions.py index a3b2a8a..979468b 100644 --- a/tests/test_decisions.py +++ b/tests/test_decisions.py @@ -105,11 +105,11 @@ def test_score_bounds(): def test_request_question_bounds(): sp.DecisionRequest.model_validate(_request([_predicate()])) - sp.DecisionRequest.model_validate(_request([_predicate()] * 200)) - with pytest.raises(ValidationError, match="1-200"): + sp.DecisionRequest.model_validate(_request([_predicate()] * 400)) + with pytest.raises(ValidationError, match="1-400"): sp.DecisionRequest.model_validate(_request([])) - with pytest.raises(ValidationError, match="1-200"): - sp.DecisionRequest.model_validate(_request([_predicate()] * 201)) + with pytest.raises(ValidationError, match="1-400"): + sp.DecisionRequest.model_validate(_request([_predicate()] * 401)) def test_request_ignores_sdk_optional_fields(): @@ -147,7 +147,116 @@ def test_input_image_rejected(): sp._decisions_input_to_state(parsed) -def test_input_non_user_role_rejected(): +def test_input_json_part_flattens_like_an_object_state(): + """A JSON part is compact JSON text, the rule an object `state` follows on the systemone route.""" + data = {"ticket": {"body": "Please reverse the second payment."}, "order": {"payments": [49, 49]}} + parsed = [ + sp.DecisionInputMessage.model_validate({"role": "user", "content": [{"type": "input_json", "data": data}]}) + ] + assert sp._decisions_input_to_state(parsed) == sp._text_of(data) + rows = [{"a": 1}, {"b": "é"}] + parsed = [ + sp.DecisionInputMessage.model_validate({"role": "user", "content": [{"type": "input_json", "data": rows}]}) + ] + assert sp._decisions_input_to_state(parsed) == '[{"a": 1}, {"b": "é"}]' + + +def test_input_json_data_must_be_an_object_or_array_of_objects(): + for bad in ("text", 3, [1, 2], None): + with pytest.raises(ValidationError): + sp.DecisionInputMessage.model_validate({"role": "user", "content": [{"type": "input_json", "data": bad}]}) + + +def test_input_json_mixed_with_text_keeps_part_order(): + parts = [{"type": "input_text", "text": "Evaluate."}, {"type": "input_json", "data": {"k": "v"}}] + parsed = [sp.DecisionInputMessage.model_validate({"role": "user", "content": parts})] + assert sp._decisions_input_to_state(parsed) == 'Evaluate.\n\n{"k": "v"}' + + +def test_input_image_with_detail_gets_the_image_message(): + data = [ + { + "role": "user", + "content": [{"type": "input_image", "image_url": "data:image/png;base64,AA", "detail": "low"}], + } + ] + parsed = [sp.DecisionInputMessage.model_validate(m) for m in data] + with pytest.raises(ValueError, match="image input is not supported"): + sp._decisions_input_to_state(parsed) + + +def test_instruction_parts_flatten_like_input_parts(): + parts = [{"type": "input_text", "text": "Is it urgent?"}, {"type": "input_json", "data": {"sla": "4h"}}] + q = sp.DecisionQuestion.model_validate(_predicate(instructions=parts)) + _, internal, _ = sp._decisions_question_to_internal(q, 0) + assert internal.instructions == 'Is it urgent?\n\n{"sla": "4h"}' + + +def test_instruction_image_part_rejected(): + parts = [ + {"type": "input_text", "text": "Same as this?"}, + {"type": "input_image", "image_url": "https://x.test/y.png"}, + ] + q = sp.DecisionQuestion.model_validate(_predicate(instructions=parts)) + with pytest.raises(ValueError, match="image input is not supported"): + sp._decisions_question_to_internal(q, 0) + + +def test_instruction_parts_reject_an_empty_list_and_cache_breakpoints(): + with pytest.raises(ValidationError): + sp.DecisionQuestion.model_validate(_predicate(instructions=[])) + part = {"type": "input_text", "text": "x", "prompt_cache_breakpoint": {"mode": "explicit"}} + with pytest.raises(ValidationError): + sp.DecisionQuestion.model_validate(_predicate(instructions=[part])) + + +def _req_input(*items): + return sp.DecisionRequest.model_validate(_request([_predicate()], input=list(items))).input + + +@pytest.mark.parametrize( + "item", + [ + {"role": "developer", "content": "Be terse."}, + {"role": "system", "content": [{"type": "input_text", "text": "Be terse."}]}, + {"role": "assistant", "content": "Let me check."}, + { + "role": "assistant", + "content": [{"type": "output_text", "text": "ok"}, {"type": "refusal", "refusal": "no"}], + }, + {"type": "function_call", "call_id": "c", "name": "f", "arguments": "{}"}, + {"type": "function_call_output", "call_id": "c", "output": "42"}, + {"type": "custom_tool_call", "call_id": "c", "name": "t", "input": "x"}, + {"type": "custom_tool_call_output", "call_id": "c", "output": [{"type": "input_text", "text": "y"}]}, + ], + ids=[ + "developer", + "system", + "assistant", + "assistant parts", + "function call", + "function output", + "custom call", + "custom output", + ], +) +def test_developer_system_assistant_and_tool_items_get_a_400_that_says_what_to_do(item): + with pytest.raises(ValidationError, match="not supported: the decision adapter reads user text"): + _req_input({"role": "user", "content": "hi"}, item) + + +def test_unknown_roles_and_item_types_are_still_rejected(): + for item in ({"role": "tool", "content": "x"}, {"type": "mcp_call", "call_id": "c"}): + with pytest.raises(ValidationError): + _req_input(item) + + +def test_user_only_input_flattens_exactly_as_before(): + items = _req_input({"role": "user", "content": "first"}, {"role": "user", "content": "second"}) + assert sp._decisions_input_to_state(items) == "first\n\nsecond" + + +def test_the_user_message_model_still_refuses_other_roles(): with pytest.raises(ValidationError): sp.DecisionInputMessage.model_validate({"role": "assistant", "content": "x"}) @@ -287,7 +396,7 @@ def test_score_answer_carries_original_labels(): @pytest.fixture() def client(monkeypatch): - async def fake_answer(name, q, state, chat, tok, served_model, adapter_name, renderer, reads=None): + async def fake_answer(name, q, state, chat, tok, served_model, adapter_name, renderer, reads=None, options=None): if q.type == "choice": keys = list(q.criteria) probs = [0.8, 0.2] + [0.0] * (len(keys) - 2) @@ -308,7 +417,12 @@ async def fake_answer(name, q, state, chat, tok, served_model, adapter_name, ren "probabilities": {str(i): 1.0 / count for i in range(count)}, "confidence": 0.3, } - return name, answer, sp.TokenUsage(input_tokens=10, output_tokens=1, cached_tokens=4) + return ( + name, + answer, + sp.TokenUsage(input_tokens=10, output_tokens=1, cached_tokens=4), + sp.QuestionDiagnostics(0.97, True), + ) monkeypatch.setattr(sp, "_answer_question", fake_answer) @@ -381,7 +495,18 @@ def test_decisions_route_validation_error_uses_openai_shape(client): ) assert resp.status_code == 400 assert resp.json()["error"]["type"] == "invalid_request_error" - assert "1-200" in resp.json()["error"]["message"] + assert "1-400" in resp.json()["error"]["message"] + + +def test_decisions_route_instruction_image_rejected_with_openai_error(client): + parts = [{"type": "input_text", "text": "Same?"}, {"type": "input_image", "image_url": "https://x.test/y.png"}] + resp = client.post( + "/v1/decisions", + json={"model": "granite-switch", "input": "x", "questions": [{"type": "predicate", "instructions": parts}]}, + ) + assert resp.status_code == 400 + assert resp.json()["error"]["type"] == "invalid_request_error" + assert "image input is not supported" in resp.json()["error"]["message"] def test_decisions_route_image_input_rejected(client): diff --git a/tests/test_openai_spec.py b/tests/test_openai_spec.py index 4a1f4de..a55c0f9 100644 --- a/tests/test_openai_spec.py +++ b/tests/test_openai_spec.py @@ -1,8 +1,8 @@ # SPDX-License-Identifier: Apache-2.0 -"""Conformance of /v1/decisions to OpenAI's published OpenAPI schema, checked with a JSON -Schema validator. The schema is fetched at test time from openai/openai-openapi (MIT) at a -pinned commit and hash; see tests/specs.py. The test skips, saying how to supply it, when -it cannot be fetched. +"""Conformance of /v1/decisions to OpenAI's published OpenAPI schema (general availability, +2026-10-22 and later), checked with a JSON Schema validator. The schema is fetched at test +time from openai/openai-openapi (MIT) at a pinned commit and hash; see tests/specs.py. The +test skips, saying how to supply it, when it cannot be fetched. Every spec-valid request must answer 200 and every 200 body must validate against `DecisionResponse`, except the deliberate limits listed in `UNSUPPORTED`, which must @@ -52,7 +52,7 @@ def _message(role, content, **fields): "empty string input": _body(input=""), "empty message list": _body(input=[]), "all three question types": _body(questions=[PREDICATE, CHOICE, SCORE]), - "200 questions": _body(questions=[PREDICATE] * 200), + "400 questions": _body(questions=[PREDICATE] * 400), "empty instructions": _body(questions=[{**PREDICATE, "instructions": ""}]), "named predicate": _body(questions=[{**PREDICATE, "name": "urgent"}]), "boolean choice values": _body(questions=[{**CHOICE, "choices": [{"value": True}, {"value": False}]}]), @@ -63,6 +63,28 @@ def _message(role, content, **fields): "user message, input_text parts": _body( input=[_message("user", [{"type": "input_text", "text": "a"}, {"type": "input_text", "text": "b"}])] ), + "input_json object part": _body( + input=[_message("user", [{"type": "input_json", "data": {"ticket": {"body": "Refund me."}, "n": [1, 2]}}])] + ), + "input_json array of objects": _body( + input=[_message("user", [{"type": "input_json", "data": [{"a": 1}, {"b": None}]}])] + ), + "text and JSON parts": _body( + input=[ + _message("user", [{"type": "input_text", "text": "Evaluate."}, {"type": "input_json", "data": {"k": 1}}]) + ] + ), + "predicate with instruction parts": _body( + questions=[ + { + **PREDICATE, + "instructions": [{"type": "input_text", "text": "Urgent?"}, {"type": "input_json", "data": {"a": 1}}], + } + ] + ), + "choice with a JSON instruction part": _body( + questions=[{**CHOICE, "instructions": [{"type": "input_json", "data": [{"policy": "refunds"}]}]}] + ), "input_text with a cache breakpoint": _body( input=[ _message("user", [{"type": "input_text", "text": "a", "prompt_cache_breakpoint": {"mode": "explicit"}}]) @@ -74,23 +96,48 @@ def _message(role, content, **fields): "null optional fields": _body(prompt_cache_key=None, prompt_cache_options=None, safety_identifier=None), } -# Spec-valid requests this text decision adapter rejects on purpose (README, "Deliberate -# scope limits on this route"). Each must fail with a 400 in OpenAI's error shape. +# Spec-valid requests this text decision adapter rejects on purpose (README, "What +# /v1/decisions doesn't support"). Each must fail with a 400 in OpenAI's error shape. UNSUPPORTED = { - "image part": _body(input=[_message("user", [{"type": "input_image", "image_url": "https://x.test/y.png"}])]), "system message": _body(input=[_message("system", "Be terse."), _message("user", "hi")]), - "developer message": _body(input=[_message("developer", "Be terse."), _message("user", "hi")]), + "developer message with text parts": _body( + input=[_message("developer", [{"type": "input_text", "text": "Be terse."}]), _message("user", "hi")] + ), "assistant message": _body(input=[_message("user", "hi"), _message("assistant", "hello")]), - "function call items": _body( + "assistant message with output_text and refusal parts": _body( + input=[ + _message("user", "hi"), + _message("assistant", [{"type": "output_text", "text": "hello"}, {"type": "refusal", "refusal": "no"}]), + ] + ), + "function call and output": _body( input=[ {"type": "function_call", "call_id": "c1", "name": "f", "arguments": "{}"}, {"type": "function_call_output", "call_id": "c1", "output": "42"}, ] ), - "custom tool call items": _body( + "custom tool call and output": _body( input=[ {"type": "custom_tool_call", "call_id": "c1", "name": "t", "input": "x"}, - {"type": "custom_tool_call_output", "call_id": "c1", "output": "y"}, + {"type": "custom_tool_call_output", "call_id": "c1", "output": [{"type": "input_text", "text": "y"}]}, + ] + ), + "image part": _body(input=[_message("user", [{"type": "input_image", "image_url": "https://x.test/y.png"}])]), + "duplicate choice values": _body(questions=[{**CHOICE, "choices": [{"value": "a"}, {"value": "a"}]}]), + "image part with detail": _body( + input=[_message("user", [{"type": "input_image", "image_url": "data:image/png;base64,AA", "detail": "low"}])] + ), + "image in question instructions": _body( + questions=[{**PREDICATE, "instructions": [{"type": "input_image", "image_url": "https://x.test/y.png"}]}] + ), + "image in a tool output": _body( + input=[ + {"type": "function_call", "call_id": "c1", "name": "f", "arguments": "{}"}, + { + "type": "function_call_output", + "call_id": "c1", + "output": [{"type": "input_image", "image_url": "https://x.test/y.png"}], + }, ] ), } @@ -99,7 +146,7 @@ def _message(role, content, **fields): "missing model": {"input": "x", "questions": [PREDICATE]}, "missing input": {"model": "m", "questions": [PREDICATE]}, "no questions": _body(questions=[]), - "201 questions": _body(questions=[PREDICATE] * 201), + "401 questions": _body(questions=[PREDICATE] * 401), "1 choice": _body(questions=[{**CHOICE, "choices": [{"value": "a"}]}]), "256 choices": _body(questions=[{**CHOICE, "choices": [{"value": f"v{i}"} for i in range(256)]}]), "1 score level": _body(questions=[{**SCORE, "levels": [{"label": "a"}]}]), @@ -107,6 +154,17 @@ def _message(role, content, **fields): "predicate without instructions": _body(questions=[{"type": "predicate"}]), "numeric choice values": _body(questions=[{**CHOICE, "choices": [{"value": 1}, {"value": 2}]}]), "level without label": _body(questions=[{**SCORE, "levels": [{"description": "a"}, {"description": "b"}]}]), + "input_json with string data": _body(input=[_message("user", [{"type": "input_json", "data": "x"}])]), + "input_json with an array of numbers": _body(input=[_message("user", [{"type": "input_json", "data": [1, 2]}])]), + "empty instructions array": _body(questions=[{**PREDICATE, "instructions": []}]), + "cache breakpoint in an instruction part": _body( + questions=[ + { + **PREDICATE, + "instructions": [{"type": "input_text", "text": "x", "prompt_cache_breakpoint": {"mode": "explicit"}}], + } + ] + ), "unknown top-level field": _body(temperature=0), "unknown question field": _body(questions=[{**PREDICATE, "extra": 1}]), "unknown message role": _body(input=[_message("tool", "x")]), @@ -117,6 +175,7 @@ def _message(role, content, **fields): # Spec-invalid requests the plugin answers anyway: the fields are accepted and ignored, # or a missing name is treated as one, so tightening them would only break clients. +# `input_json` over OpenAI's stated size limits is also accepted; this server's own limits apply. ACCEPTED_LENIENTLY = { "null question name": _body(questions=[{**PREDICATE, "name": None}]), "unsupported cache ttl": _body(prompt_cache_options={"ttl": "1h"}), @@ -130,7 +189,7 @@ def _errors(validator, instance): @pytest.fixture() def client(monkeypatch): - async def fake_answer(name, q, state, chat, tok, served_model, adapter_name, renderer, reads=None): + async def fake_answer(name, q, state, chat, tok, served_model, adapter_name, renderer, reads=None, options=None): if q.type == "choice": keys = list(q.criteria) answer = { @@ -150,7 +209,12 @@ async def fake_answer(name, q, state, chat, tok, served_model, adapter_name, ren "probabilities": {str(i): 1.0 / n for i in range(n)}, "confidence": 0.3, } - return name, answer, sp.TokenUsage(input_tokens=10, output_tokens=1, cached_tokens=4) + return ( + name, + answer, + sp.TokenUsage(input_tokens=10, output_tokens=1, cached_tokens=4), + sp.QuestionDiagnostics(0.97, True), + ) monkeypatch.setattr(sp, "_answer_question", fake_answer) @@ -205,6 +269,35 @@ def test_lenient_acceptance_still_returns_a_spec_valid_response(client, name): assert not _errors(RESPONSE, resp.json()) +# vLLM's own /v1/decisions takes these two fields beyond the schema (cache_salt, joint). +VLLM_EXTENSIONS = { + "cache salt": _body(cache_salt="tenant-1"), + "joint false": _body(joint=False), +} + + +@pytest.mark.parametrize("name", VLLM_EXTENSIONS) +def test_vllm_extension_is_accepted_with_a_spec_valid_response(client, name): + body = VLLM_EXTENSIONS[name] + assert _errors(REQUEST, body), "test case is spec-valid, so it is not an extension" + resp = client.post("/v1/decisions", json=body) + assert resp.status_code == 200, resp.text + assert not _errors(RESPONSE, resp.json()) + + +def test_the_response_has_no_diagnostics_because_the_schema_forbids_extra_fields(client): + data = client.post("/v1/decisions", json=_body()).json() + assert "diagnostics" not in data and not _errors(RESPONSE, data) + + +@pytest.mark.parametrize("field", ["instructions", "chat_template_kwargs", "priority", "request_id"]) +def test_systemone_only_fields_are_rejected_on_decisions(client, field): + value = {"chat_template_kwargs": {}, "priority": 0}.get(field, "x") + resp = client.post("/v1/decisions", json=_body(**{field: value})) + assert resp.status_code == 400 + assert resp.json()["error"]["type"] == "invalid_request_error" + + def test_usage_reports_cached_tokens_summed_over_questions(client): resp = client.post("/v1/decisions", json=_body(questions=[PREDICATE, CHOICE, SCORE])) usage = resp.json()["usage"] diff --git a/tests/test_plugin.py b/tests/test_plugin.py index 550b8b5..0b5f0b9 100644 --- a/tests/test_plugin.py +++ b/tests/test_plugin.py @@ -482,7 +482,7 @@ def test_answer_question_counts_every_chunk_of_a_chunked_prompt( each call prefills (or reads from cache) its own copy of the prompt.""" calls = [] - async def fake_fetch(chat, served_model, adapter_name, msgs, label_ids_chunk): + async def fake_fetch(chat, served_model, adapter_name, msgs, label_ids_chunk, cache_salt=None): calls.append(label_ids_chunk) cached, written = first_chunk_cache if len(calls) == 1 else (100, 0) return _fake_chat_response(label_ids_chunk, 100, cached, written) @@ -492,7 +492,7 @@ async def fake_fetch(chat, served_model, adapter_name, msgs, label_ids_chunk): q = sp.Question(type="choice", instructions="Which?", criteria=criteria) renderer = lambda tok, state, instructions, descs: ([], list(range(1000, 1000 + len(descs)))) # noqa: E731 - _, answer, usage = asyncio.run( + _, answer, usage, _ = asyncio.run( sp._answer_question("q", q, "state", None, None, "m", "decision", renderer, asyncio.Semaphore(8)) ) @@ -590,7 +590,7 @@ def renderer(tok, state, instructions, descs): seen["descs"] = descs return [], list(range(len(descs))) - async def fake_fetch(chat, served_model, adapter_name, msgs, label_ids_chunk): + async def fake_fetch(chat, served_model, adapter_name, msgs, label_ids_chunk, cache_salt=None): return _fake_chat_response(label_ids_chunk, 10, None, None) monkeypatch.setattr(sp, "_fetch_label_logprobs", fake_fetch) @@ -711,7 +711,7 @@ def test_engine_error_status_need_not_be_a_standard_http_status(): def test_missing_label_logprob_is_an_engine_error(monkeypatch): - async def fake_fetch(chat, served_model, adapter_name, msgs, label_ids_chunk): + async def fake_fetch(chat, served_model, adapter_name, msgs, label_ids_chunk, cache_salt=None): return _fake_chat_response(label_ids_chunk[:-1], 10, None, None) monkeypatch.setattr(sp, "_fetch_label_logprobs", fake_fetch) @@ -726,7 +726,7 @@ def test_a_failing_label_chunk_cancels_its_sibling(monkeypatch): running (asyncio.run would cancel it at shutdown anyway, so check inside the loop).""" sibling = {} - async def fake_fetch(chat, served_model, adapter_name, msgs, label_ids_chunk): + async def fake_fetch(chat, served_model, adapter_name, msgs, label_ids_chunk, cache_salt=None): if label_ids_chunk[0] == 0: await asyncio.sleep(0) raise sp.EngineError("boom", 500) @@ -748,7 +748,7 @@ async def run(): def test_engine_reads_per_request_are_bounded(monkeypatch): in_flight, peak = [0], [0] - async def fake_fetch(chat, served_model, adapter_name, msgs, label_ids_chunk): + async def fake_fetch(chat, served_model, adapter_name, msgs, label_ids_chunk, cache_salt=None): in_flight[0] += 1 peak[0] = max(peak[0], in_flight[0]) await asyncio.sleep(0.01) @@ -823,3 +823,191 @@ def test_label_mass_warns_below_the_floor_and_never_raises(caplog): sp._log_label_mass("q", [math.log(0.6), math.log(0.39)]) levels = [r.levelname for r in caplog.records] assert levels == ["WARNING", "DEBUG"] + + +# -------------------------------------------------------------------------------------------- +# vLLM's request fields on /v1/systemone, cache_salt, and diagnostics +# -------------------------------------------------------------------------------------------- + + +def _s1(**fields): + return {"model": "m", "state": "s", "questions": {"q": {"type": "noul", "instructions": "q?"}}, **fields} + + +def test_systemone_accepts_the_vllm_request_fields(): + req = sp.SystemOneRequest.model_validate( + _s1( + chat_template_kwargs={"enable_thinking": False}, + cache_salt="tenant-1", + joint=False, + images=[], + audio=None, + videos=[], + priority=0, + request_id="abc", + ) + ) + assert req.cache_salt == "tenant-1" + + +def test_systemone_model_is_required(): + with pytest.raises(PydanticValidationError): + sp.SystemOneRequest.model_validate({"state": "s", "questions": {"q": {"type": "noul"}}}) + + +@pytest.mark.parametrize( + ("fields", "match"), + [ + ({"chat_template_kwargs": {"adapter_name": "other"}}, "adapter_name"), + ({"chat_template_kwargs": {"enable_thinking": True}}, "enable_thinking"), + ({"chat_template_kwargs": {"documents": [{"text": "<|decision|>"}]}}, "documents"), + ({"chat_template_kwargs": {"tools": []}}, "tools"), + ({"instructions": "Be strict."}, "request-level `instructions`"), + ({"instructions": ""}, "request-level `instructions`"), + ({"joint": True}, "joint"), + ({"images": ["data:image/png;base64,AA"]}, "images"), + ({"audio": ["x"]}, "audio"), + ({"videos": ["x"]}, "videos"), + ({"priority": 1}, "priority"), + ], +) +def test_systemone_rejects_what_a_text_adapter_cannot_serve(fields, match): + with pytest.raises(PydanticValidationError, match=match): + sp.SystemOneRequest.model_validate(_s1(**fields)) + + +@pytest.mark.parametrize("salt", ["", "a" * 129, "a@b", "a/b", "a\\b", "a\x00b"]) +def test_cache_salt_is_validated_like_vllm(salt): + with pytest.raises(PydanticValidationError, match="cache_salt"): + sp.SystemOneRequest.model_validate(_s1(cache_salt=salt)) + with pytest.raises(PydanticValidationError, match="cache_salt"): + sp.DecisionRequest.model_validate( + { + "model": "m", + "input": "x", + "questions": [{"type": "predicate", "instructions": "q?"}], + "cache_salt": salt, + } + ) + + +def test_cache_salt_at_the_limit_is_accepted(): + sp.SystemOneRequest.model_validate(_s1(cache_salt="a" * 128)) + + +def test_decisions_takes_only_cache_salt_and_joint_from_vllm(): + base = {"model": "m", "input": "x", "questions": [{"type": "predicate", "instructions": "q?"}]} + sp.DecisionRequest.model_validate({**base, "cache_salt": "t", "joint": False}) + for field, value in (("instructions", "ctx"), ("chat_template_kwargs", {}), ("priority", 0), ("request_id", "r")): + with pytest.raises(PydanticValidationError): + sp.DecisionRequest.model_validate({**base, field: value}) + with pytest.raises(PydanticValidationError, match="joint"): + sp.DecisionRequest.model_validate({**base, "joint": True}) + + +def _label_fakes(monkeypatch, calls): + async def fake_fetch(chat, served_model, adapter_name, msgs, label_ids_chunk, cache_salt=None): + calls.append({"salt": cache_salt, "adapter": adapter_name, "ids": label_ids_chunk, "msgs": msgs}) + return _fake_chat_response(label_ids_chunk, 100, None, None) + + monkeypatch.setattr(sp, "_fetch_label_logprobs", fake_fetch) + + +def test_cache_salt_reaches_every_chunk_read(monkeypatch): + calls = [] + _label_fakes(monkeypatch, calls) + q = sp.Question(type="choice", instructions="Which?", criteria={f"o{i}": None for i in range(200)}) + renderer = lambda tok, state, instructions, descs, **kw: ([], list(range(len(descs)))) # noqa: E731 + asyncio.run( + sp._answer_question( + "q", q, "s", None, None, "m", "decision", renderer, asyncio.Semaphore(8), sp.ReadOptions("tenant-1") + ) + ) + assert len(calls) == 2 and {c["salt"] for c in calls} == {"tenant-1"} + + +def test_the_servers_adapter_is_the_only_one_ever_requested(monkeypatch): + """The engine request is built from the server's adapter name and the request's salt alone. + Caller `chat_template_kwargs` are validated and dropped, so they cannot reach it.""" + import sys + from types import ModuleType, SimpleNamespace + + captured = {} + + class FakeRequest: + def __init__(self, **kw): + captured.update(kw) + + class FakeResponse: + pass + + class FakeChat: + async def create_chat_completion(self, req): + return FakeResponse() + + protocol = ModuleType("vllm.entrypoints.openai.chat_completion.protocol") + protocol.ChatCompletionRequest, protocol.ChatCompletionResponse = FakeRequest, FakeResponse + serve = ModuleType("vllm.entrypoints.serve.engine.protocol") + serve.ErrorResponse = type("ErrorResponse", (), {}) + for name, mod in ( + ("vllm.entrypoints.openai.chat_completion.protocol", protocol), + ("vllm.entrypoints.serve.engine.protocol", serve), + ): + monkeypatch.setitem(sys.modules, name, mod) + for parent in ( + "vllm", + "vllm.entrypoints", + "vllm.entrypoints.openai", + "vllm.entrypoints.openai.chat_completion", + "vllm.entrypoints.serve", + "vllm.entrypoints.serve.engine", + ): + monkeypatch.setitem(sys.modules, parent, sys.modules.get(parent) or ModuleType(parent)) + + asyncio.run(sp._fetch_label_logprobs(FakeChat(), "granite-switch", "decision", [], [1, 2], "tenant-1")) + assert captured["chat_template_kwargs"] == {"adapter_name": "decision"} + assert captured["cache_salt"] == "tenant-1" + asyncio.run(sp._fetch_label_logprobs(FakeChat(), "granite-switch", "decision", [], [1, 2])) + assert captured["cache_salt"] is None + assert SimpleNamespace # keep the import used + + +def test_a_caller_cannot_name_the_adapter_in_template_kwargs(): + with pytest.raises(PydanticValidationError, match="set by the server"): + sp.SystemOneRequest.model_validate(_s1(chat_template_kwargs={"adapter_name": "guardian-core"})) + + +def test_answer_question_reports_label_mass_and_argmax(monkeypatch): + async def fake_fetch(chat, served_model, adapter_name, msgs, label_ids_chunk, cache_salt=None): + resp = _fake_chat_response(label_ids_chunk, 10, None, None) + resp.choices[0].logprobs.content[0].token = f"token_id:{label_ids_chunk[1]}" + resp.choices[0].logprobs.content[0].top_logprobs = [ + type(t)(token=t.token, logprob=lp) + for t, lp in zip(resp.choices[0].logprobs.content[0].top_logprobs, (math.log(0.2), math.log(0.7))) + ] + return resp + + monkeypatch.setattr(sp, "_fetch_label_logprobs", fake_fetch) + q = sp.Question(type="noul", instructions="q?") + renderer = lambda tok, state, instructions, descs, **kw: ([], [10, 11]) # noqa: E731 + _, answer, _, diagnostics = asyncio.run( + sp._answer_question("q", q, "s", None, None, "m", "decision", renderer, asyncio.Semaphore(8)) + ) + assert diagnostics.label_mass == pytest.approx(0.9) + assert diagnostics.argmax_is_label is True + assert answer["noul"] == pytest.approx(0.7 / 0.9) + + +def test_argmax_is_not_a_label_when_the_engine_sampled_something_else(monkeypatch): + async def fake_fetch(chat, served_model, adapter_name, msgs, label_ids_chunk, cache_salt=None): + resp = _fake_chat_response(label_ids_chunk, 10, None, None) + resp.choices[0].logprobs.content[0].token = "token_id:999" + return resp + + monkeypatch.setattr(sp, "_fetch_label_logprobs", fake_fetch) + q = sp.Question(type="noul", instructions="q?") + renderer = lambda tok, state, instructions, descs, **kw: ([], [10, 11]) # noqa: E731 + *_, diagnostics = asyncio.run( + sp._answer_question("q", q, "s", None, None, "m", "decision", renderer, asyncio.Semaphore(8)) + ) + assert diagnostics.argmax_is_label is False diff --git a/tests/test_systemone_spec.py b/tests/test_systemone_spec.py index 368a978..4038657 100644 --- a/tests/test_systemone_spec.py +++ b/tests/test_systemone_spec.py @@ -46,10 +46,27 @@ def _body(**fields): "score with 10 levels": _body(questions={"s": {"type": "score", "criteria": [str(i) for i in range(10)]}}), "object criteria values": _body(questions={"c": {"type": "choice", "criteria": {"a": {"text": "A"}, "b": ["B"]}}}), } +# vLLM's own /v1/systemone fields. jev's schema does not have them, so the plugin accepts them as +# an extension (the responses must still be valid jev). +VLLM_FIELDS = { + "chat_template_kwargs": _body(chat_template_kwargs={"enable_thinking": False}), + "cache_salt": _body(cache_salt="tenant-1"), + "joint false": _body(joint=False), + "empty media": _body(images=[], audio=[], videos=[]), + "priority 0 and request id": _body(priority=0, request_id="r1"), +} # Valid in jev's format but rejected on purpose: no image support (`box`), and unknown # fields, which jev's schema allows and this plugin refuses. REJECTED = { "box question": _body(questions={"x": {"type": "box", "instructions": "Find the cat"}}), + "request-level instructions": _body(instructions="Be strict."), + "adapter name in template kwargs": _body(chat_template_kwargs={"adapter_name": "guardian-core"}), + "unsupported template kwarg": _body(chat_template_kwargs={"documents": []}), + "thinking on": _body(chat_template_kwargs={"enable_thinking": True}), + "joint true": _body(joint=True), + "bad cache salt": _body(cache_salt="a/b"), + "an image": _body(images=["data:image/png;base64,AA"]), + "non-zero priority": _body(priority=3), "unknown top-level field": _body(temperature=0), "unknown question field": _body(questions={"q": {**N, "extra": 1}}), } @@ -78,7 +95,7 @@ def _errors(validator, instance): @pytest.fixture() def client(monkeypatch): - async def fake_answer(name, q, state, chat, tok, served_model, adapter_name, renderer, reads=None): + async def fake_answer(name, q, state, chat, tok, served_model, adapter_name, renderer, reads=None, options=None): if q.type == "choice": keys = list(q.criteria) answer = { @@ -98,7 +115,7 @@ async def fake_answer(name, q, state, chat, tok, served_model, adapter_name, ren "probabilities": {str(i): 1.0 / n for i in range(n)}, "confidence": 0.3, } - return name, answer, sp.TokenUsage(input_tokens=10, output_tokens=1) + return name, answer, sp.TokenUsage(input_tokens=10, output_tokens=1), sp.QuestionDiagnostics(0.97, True) monkeypatch.setattr(sp, "_answer_question", fake_answer) @@ -150,3 +167,32 @@ def test_request_outside_the_published_bounds_gets_a_valid_422(client, name): resp = client.post("/v1/systemone", json=OUT_OF_BOUNDS[name]) assert resp.status_code == 422, resp.text assert not _errors(ERROR, resp.json()) + + +@pytest.mark.parametrize("name", VLLM_FIELDS) +def test_vllm_fields_are_accepted_and_the_response_is_still_valid_jev(client, name): + resp = client.post("/v1/systemone", json=VLLM_FIELDS[name]) + assert resp.status_code == 200, resp.text + assert not _errors(RESPONSE, resp.json()) + + +def test_response_carries_diagnostics_for_every_question(client): + body = _body(questions={"n": N, "c": C, "s": S}) + data = client.post("/v1/systemone", json=body).json() + assert set(data["diagnostics"]) == {"n", "c", "s"} + assert data["diagnostics"]["n"] == {"label_mass": 0.97, "argmax_is_label": True} + assert not _errors(RESPONSE, data) + + +def test_the_salt_reaches_the_engine_options(client, monkeypatch): + seen = {} + real = sp._answer_all + + async def spy(state, state_value, questions, options=sp.ReadOptions()): + seen["options"] = options + return await real(state, state_value, questions, options) + + monkeypatch.setattr(sp, "_answer_all", spy) + resp = client.post("/v1/systemone", json=_body(cache_salt="tenant-1")) + assert resp.status_code == 200 + assert seen["options"] == sp.ReadOptions("tenant-1")