feat: conform /v1/decisions to the GA Decisions schema and align with vLLM's request fields

CONTRIBUTING.md

@@ -67,8 +67,8 @@ unsigned commits fail before they are pushed.
 - New behaviour needs a test. [docs/TESTING.md](docs/TESTING.md) records what each
   existing test covers
 - Wire-format behaviour is pinned against the published OpenAI and TypeSafe schemas;
-  if you change a route's shape, update the spec tests and the deliberate-exceptions
-  list in the README together
+  if you change a route's shape, update the spec tests and the README's list of
+  unsupported requests together
 
 ## Code of Conduct
 

README.md

@@ -4,13 +4,16 @@ Decision APIs for [Granite Switch](https://github.com/generative-computing/grani
 vLLM plugin. Ask a question about some text and get back probabilities, from the same server and
 GPU cache that already serves `/v1/chat/completions`.
 
-- `POST /v1/decisions` follows [OpenAI's Decisions API](https://developers.openai.com/api/docs/guides/decisions).
-  The official OpenAI SDKs work once you change `base_url`.
+- `POST /v1/decisions` follows [OpenAI's Decisions API](https://developers.openai.com/api/docs/guides/decisions)
+  for text input, and the official OpenAI SDKs work once you change `base_url`. A few parts of the API aren't
+  supported, and those requests get a clear error ([details](#what-v1decisions-doesnt-support)).
 - `POST /v1/systemone` follows [jev's system-one API](https://docs.typesafe.ai/api).
 
 Each question is answered with one engine read of the adapter's single-token option labels, so
 callers do not need a prompt format, an invocation marker or a label scheme.
 
+With the public Granite Switch checkpoints, results are comparable with those on their model cards.
+
 You can run it as a container or add it to a vLLM install you already have. Start with
 [Quick start](#quick-start). What has and has not been tested is in
 [docs/STATUS.md](docs/STATUS.md), and how to run the tests is in [docs/TESTING.md](docs/TESTING.md).
@@ -109,6 +112,25 @@ curl http://localhost:8000/v1/decisions -H "Content-Type: application/json" -d '
 equivalent `/v1/systemone` request. For equivalent questions both routes build the same prompt and
 return the same probabilities.
 
+### More request options
+
+`/v1/systemone` also accepts the extra fields on vLLM's own endpoint, so most clients written for vLLM work as they
+are.
+
+- **`cache_salt`** keeps a request's prefix cache separate from everyone else's. The cache is shared, so a fast
+  response (or a `cached_tokens` count) can show that someone has already sent the same prompt. If you serve several
+  tenants, have your gateway set a long random `cache_salt` for each. It's a privacy measure, not authentication.
+  `/v1/decisions` accepts it too.
+- **`chat_template_kwargs`** may only be `{"enable_thinking": false}`. The server always chooses the adapter.
+- **`joint`, `images`, `audio`, `videos`, `priority`, `request_id`** are accepted at their defaults. This model reads
+  text, one question at a time, so other values get a 422.
+- **`model`** is required here, though vLLM makes it optional.
+
+Each question in a `/v1/systemone` response also carries `diagnostics`: `label_mass`, how much probability the model
+put on the option labels (a low value usually means a wrong marker or layout), and `argmax_is_label`. OpenAI's
+response schema doesn't allow extra fields, so `/v1/decisions` has none. `confidence` uses jev's formula, so it isn't
+comparable with vLLM's.
+
 ## Secure it before you expose it
 
 The server listens on `0.0.0.0` and, as shipped, asks for no credentials on any route. Put it behind a
@@ -219,10 +241,11 @@ same prompt, served from the prefix cache.
 - **Caller text is escaped.** Special tokens and the invocation marker inside the state, instructions
   or descriptions are broken with an invisible U+2060, so they cannot forge turns or move the adapter's
   activation point.
-- **Input.** `input` messages must have the role `user`, and images are rejected, because the trained
-  layout has one text state turn. `/v1/systemone` rejects unknown fields and `box` questions.
+- **Input.** `input` is a string or a list of user messages. A message can hold text and `input_json` parts, which
+  are written as compact JSON, as an object `state` is on `/v1/systemone`. A question's `instructions` can be a
+  string or text and JSON parts. `/v1/systemone` rejects unknown fields and `box` questions.
 - **Size.** The state, instructions and options of each question are counted before any engine work:
-  at most 200 questions, 32k tokens per question and 64k per request (262k with
+  at most 400 questions, 32k tokens per question and 64k per request (262k with
   `TypeSafe-Danger-Zone-Long-Context: true` on `/v1/systemone`), all capped by `--max-model-len`.
   Bodies over 32 MiB, nesting deeper than 64 and non-finite numbers are rejected.
 - **Usage.** A `choice` with more than 128 options runs as two engine requests over the same prompt,
@@ -235,9 +258,22 @@ same prompt, served from the prefix cache.
 - **Cancellation.** A client that disconnects cancels its request's engine work.
 - **Errors.** `/v1/decisions` uses OpenAI's error shape and `/v1/systemone` uses jev's
   `HTTPValidationError` for 422s. Engine errors keep vLLM's status code.
+- **Repeat requests.** The same request can return slightly different probabilities, depending on how it is
+  batched and whether it hits the prefix cache, so a near-tie can flip. A different GPU or tensor-parallel size can
+  differ a little more. vLLM's `VLLM_BATCH_INVARIANT=1` aims to make repeats exact; we haven't tried it here.
 - **Not provided.** Authentication (use vLLM's `--api-key`), rate limiting and `refusal` answers.
-  Engine reads are not bit-identical between batched and solo requests, so repeated calls can differ in
-  the third decimal place.
+
+### What /v1/decisions doesn't support
+
+The decision adapter reads one block of text and a question, and this server keeps to that layout so results stay
+predictable. So a few requests that OpenAI's schema allows get a 400 that names the part:
+
+- developer, system and assistant messages (put what the questions are about in a user message)
+- tool calls and tool outputs
+- images, in `input` or in a question's `instructions`
+- choices that would merge into one: two with the same value, or `"true"` and `true`
+
+The route never returns `refusal` answers, and `/v1/systemone` likewise refuses request-level `instructions`.
 
 ## Upstream status
 

docs/STATUS.md

@@ -15,14 +15,14 @@ and spec suites and the 8b battery (`e2e_check`, `probe_check`, `accuracy_check`
 | Area | How | Result |
 |---|---|---|
 | Unit and route tests | `pytest`, engine stubbed, in CI | pass |
-| `/v1/decisions` against OpenAI's OpenAPI schema | `tests/test_openai_spec.py` (41 cases), plus the same request set live | pass, with the exceptions listed in the README |
+| `/v1/decisions` against OpenAI's OpenAPI schema | `tests/test_openai_spec.py` (against the general-availability schema), plus the same request set live | pass, with the exceptions listed in the README |
 | `/v1/systemone` against jev's wire format | `tests/test_systemone_spec.py` (24 cases) against `typesafe-sdk`'s generated models, stubbed | pass; the plugin also rejects unknown fields, which jev's format allows |
-| Live wire battery | `scripts/e2e_check.py` (11 checks) | 11/11 |
+| Live wire battery | `scripts/e2e_check.py` (12 checks, including vLLM's request fields and `cache_salt`) | 12/12 |
 | Prompt rendering | server prompt token counts against an offline render with the checkpoint's tokenizer and chat template | identical |
 | Prefix-cache sharing | `tests/test_prefix_sharing.py`, and `cached_tokens` live, cross-checked with `/metrics` | as documented in the README |
 | Route parity | 800 labelled items through `/v1/decisions` and through the same prompt over chat completions | no item answers differently |
 | Escaping, size guard, disconnect cancellation, error shapes | live probes, now `scripts/probe_check.py` | as documented in the README |
-| Question cap and `DECISION_MAX_CONCURRENT_REQUESTS` | live: 200 questions accepted, 201 rejected; a 32-request burst against a cap of 8 gave 8 accepted and 24 rejected with 503 | pass |
+| Question cap and `DECISION_MAX_CONCURRENT_REQUESTS` | live: 400 questions accepted, 401 rejected; a 32-request burst against a cap of 8 gave 8 accepted and 24 rejected with 503 | pass |
 | `DECISION_EXPERIMENTAL_SYSTEM_PROMPT=1` | live: the 800-item comparison and `e2e_check.py --system-prompt` | answers identical to the same prompt over chat completions; decisions about one state still share the cache, chat no longer does |
 | FlashAttention 3 | FA2 and FA3 servers side by side: the 800-item comparison, 40 cached one-token reads (the case in granite-switch#139), load at 8 and 32 clients | no difference beyond batching noise in answers or speed; FA2 kept in `serve.sh` while #139 is open |
 | Load | 1 to 128 concurrent clients, three-question requests | about 140 requests/s plateau, no errors |
@@ -72,6 +72,9 @@ show that the serving path is identical to the one that produced the card's figu
 
 ## Known gaps
 
+- **`/v1/decisions` covers OpenAI's schema for text only.** Images, developer, system and assistant
+  messages and tool calls get a 400, and `/v1/systemone` refuses request-level `instructions`. The adapter
+  reads one state turn and a question, and nothing else is known to work. The README lists each case.
 - **Speculative decoding is untested and may crash the engine.** Decision reads use
   `logprob_token_ids`; vLLM issue #42592 (fix #44727, open) crashes on that whenever a
   speculative-decoding request shares the batch. We have not run MTP, so whether the decision

docs/TESTING.md

@@ -35,10 +35,12 @@ it needs `gitleaks` on `PATH` for the last of those.
   question) and error shapes.
 - `tests/test_openai_spec.py`: `/v1/decisions` requests and responses validated with a
   JSON Schema validator against OpenAI's published schema. The schema is downloaded on
-  first run from `openai/openai-openapi` (MIT) at a pinned commit, checked against a
-  pinned SHA-256 and cached in `~/.cache/granite-switch-decision`; it is never committed.
+  first run from `openai/openai-openapi` (MIT) at a pinned commit (the general-availability
+  revision), checked against a pinned SHA-256 and cached in `~/.cache/granite-switch-decision`;
+  it is never committed. To move the pin, set `OPENAI_COMMIT` in `tests/specs.py` and compute
+  `OPENAI_SHA256` with `curl -fsSL https://raw.githubusercontent.com/openai/openai-openapi/<sha>/openapi.json | shasum -a 256`.
   Offline, the test skips and says how to supply a local copy (`DECISION_OPENAI_SPEC`).
-  Deliberate limits are listed as named exceptions.
+  Unsupported requests are listed as named cases, matching the README's "What /v1/decisions doesn't support".
 - `tests/test_systemone_spec.py`: `/v1/systemone` validated against jev's wire models from
   TypeSafe's MIT-licensed `typesafe-sdk` (a dev dependency), which TypeSafe generates
   from its API spec. Those models omit the spec's bounds on choice options (1-255) and
@@ -68,7 +70,7 @@ Python standard library unless noted.
 
 | Command | What it checks |
 |---|---|
-| `python3 scripts/e2e_check.py --base-url URL` | Both routes' answer shapes and parity, input flattening, boolean values, error shapes, usage, chat on the same server, prefix-cache hits. Add `--system-prompt` when the server runs with `DECISION_EXPERIMENTAL_SYSTEM_PROMPT=1`. |
+| `python3 scripts/e2e_check.py --base-url URL` | Both routes' answer shapes and parity, input flattening, error shapes, usage, vLLM's request fields and `cache_salt`, chat on the same server, prefix-cache hits. Add `--system-prompt` when the server runs with `DECISION_EXPERIMENTAL_SYSTEM_PROMPT=1`. |
 | `python3 scripts/probe_check.py --base-url URL` | Question cap, size guard (no engine work for oversized requests), body limits, special-token and marker escaping, options without descriptions, disconnect cancellation. |
 | `python3 scripts/probe_check.py --base-url URL --max-concurrent N` | Adds the concurrency limit: a burst against `DECISION_MAX_CONCURRENT_REQUESTS=N` gets 200s and 503s. |
 | `python3 scripts/probe_check.py --base-url URL --load 30` | Adds a load test at 1, 8 and 32 clients with chat latency alongside. Reported, not asserted. |

scripts/e2e_check.py

@@ -19,12 +19,19 @@
   parity              the same questions through both routes agree: same
                       choice pick, same per-option/level probabilities, same
                       predicate probability, same score
-  input_flattening    a message-list input answers identically to the string
+  input_flattening    a message-list input, an input_json part and instruction parts
+                      answer identically to their plain-string equivalents
   boolean_values      boolean choice values round-trip as booleans
   usage               usage is present and non-zero
   errors              400s with OpenAI's error shape (empty questions, image
                       input, 11 score levels) and jev's 422 shape on the
                       systemone route
+  vllm_fields         vLLM's request fields on /v1/systemone: enable_thinking false
+                      accepted; request-level instructions, adapter_name, other
+                      template kwargs, a bad cache_salt, joint and images refused;
+                      system messages refused on /v1/decisions; a
+                      cache_salt gives its own prefix cache (needs
+                      --enable-prompt-tokens-details)
   chat                /v1/chat/completions still works on the shared server
                       (skip with --skip-chat)
   prefix_cache        prefix-cache hits for a decision after a chat or a
@@ -198,6 +205,12 @@ def check_systemone_shapes(client: Client, model: str) -> Check:
     check.expect_equal(severity.get("type"), "score", "severity type")
     check.expect(isinstance(severity.get("score"), (int, float)), "severity score")
     check.expect_equal(len(severity.get("probabilities", {})), 3, "severity probabilities")
+    diagnostics = body.get("diagnostics", {})
+    check.expect(set(diagnostics) == set(answers), "diagnostics keys match the answers")
+    for name, entry in diagnostics.items():
+        mass = entry.get("label_mass")
+        check.expect(isinstance(mass, (int, float)) and 0.0 <= mass <= 1.0 + 1e-6, f"{name} label_mass in [0, 1]")
+        check.expect(isinstance(entry.get("argmax_is_label"), bool), f"{name} argmax_is_label is a boolean")
     return check
 
 
@@ -269,6 +282,40 @@ def check_input_flattening(client: Client, model: str, tol: float) -> Check:
         "string vs message-list probability",
         tol,
     )
+
+    # A JSON part is the compact JSON text of an object state, and instruction parts join like input parts.
+    obj = {"ticket": "My payouts have failed three times.", "attempts": 3}
+    _, text_form = client.post(
+        "/v1/decisions", {"model": model, "input": json.dumps(obj, ensure_ascii=False), "questions": [question]}
+    )
+    _, json_form = client.post(
+        "/v1/decisions",
+        {
+            "model": model,
+            "input": [{"role": "user", "content": [{"type": "input_json", "data": obj}]}],
+            "questions": [question],
+        },
+    )
+    check.expect_close(
+        text_form["answers"][0]["probability"],
+        json_form["answers"][0]["probability"],
+        "JSON text vs input_json probability",
+        tol,
+    )
+    parts = [{"type": "input_text", "text": "Is this about"}, {"type": "input_text", "text": "payouts?"}]
+    _, plain = client.post(
+        "/v1/decisions",
+        {"model": model, "input": STATE, "questions": [{**question, "instructions": "Is this about\n\npayouts?"}]},
+    )
+    _, split = client.post(
+        "/v1/decisions", {"model": model, "input": STATE, "questions": [{**question, "instructions": parts}]}
+    )
+    check.expect_close(
+        plain["answers"][0]["probability"],
+        split["answers"][0]["probability"],
+        "string vs instruction-parts probability",
+        tol,
+    )
     return check
 
 
@@ -351,6 +398,57 @@ def check_errors(client: Client, model: str) -> Check:
     return check
 
 
+def check_vllm_fields(client: Client, model: str) -> Check:
+    """vLLM's own request fields: accepted where the plugin supports them, refused (and never
+    forwarded) where it cannot, and `cache_salt` really partitions the prefix cache."""
+    check = Check("vllm_fields")
+    body = {**SYSTEMONE_BODY, "model": model}
+    status, _ = client.post("/v1/systemone", {**body, "chat_template_kwargs": {"enable_thinking": False}})
+    check.expect_equal(status, 200, "enable_thinking: false")
+    status, _ = client.post("/v1/systemone", {**body, "instructions": "Answer strictly."})
+    check.expect_equal(status, 422, "request-level instructions")
+    status, _ = client.post("/v1/systemone", {**body, "chat_template_kwargs": {"adapter_name": "guardian-core"}})
+    check.expect_equal(status, 422, "adapter_name in chat_template_kwargs")
+    status, _ = client.post("/v1/systemone", {**body, "chat_template_kwargs": {"documents": [{"text": "x"}]}})
+    check.expect_equal(status, 422, "unsupported chat_template_kwargs key")
+    status, _ = client.post("/v1/systemone", {**body, "cache_salt": "a/b"})
+    check.expect_equal(status, 422, "cache_salt with a forbidden character")
+    status, _ = client.post("/v1/systemone", {**body, "joint": True})
+    check.expect_equal(status, 422, "joint: true")
+    status, _ = client.post("/v1/systemone", {**body, "images": ["data:image/png;base64,AA"]})
+    check.expect_equal(status, 422, "images")
+    status, _ = client.post("/v1/decisions", {**DECISIONS_BODY, "model": model, "instructions": "x"})
+    check.expect_equal(status, 400, "instructions on /v1/decisions")
+    system = [{"role": "system", "content": "Be terse."}, {"role": "user", "content": STATE}]
+    status, _ = client.post("/v1/decisions", {**DECISIONS_BODY, "model": model, "input": system})
+    check.expect_equal(status, 400, "system message on /v1/decisions")
+
+    # A fresh salt starts with an empty cache, the same salt then hits it, and another salt (or none) does not.
+    text = f"Ticket {uuid.uuid4().hex}. " + TICKET * 6
+    salted = {
+        "model": model,
+        "input": text,
+        "cache_salt": f"salt-{uuid.uuid4().hex}",
+        "questions": [{"type": "predicate", "instructions": "Does this need urgent attention?"}],
+    }
+
+    def cached(request: dict) -> int:
+        _, response = client.post("/v1/decisions", request)
+        return response.get("usage", {}).get("input_tokens_details", {}).get("cached_tokens", 0)
+
+    first, second = cached(salted), cached(salted)
+    other = cached({**salted, "cache_salt": f"salt-{uuid.uuid4().hex}"})
+    unsalted = cached({k: v for k, v in salted.items() if k != "cache_salt"})
+    check.notes.append(
+        f"cache_salt: first {first}, same salt again {second}, other salt {other}, no salt {unsalted} cached tokens"
+    )
+    check.expect(first == 0, f"a fresh salt read {first} tokens from the cache")
+    check.expect(second > 0, "the same salt did not hit the cache")
+    check.expect(other == 0, f"another salt read {other} tokens from the cache")
+    check.expect(unsalted == 0, f"no salt read {unsalted} tokens from the salted cache")
+    return check
+
+
 def check_chat(client: Client, model: str) -> Check:
     check = Check("chat")
     status, body = client.post(
@@ -535,6 +633,7 @@ def main() -> int:
         check_boolean_values(client, model),
         check_usage(client, model),
         check_errors(client, model),
+        check_vllm_fields(client, model),
     ]
     if not args.skip_chat:
         checks.append(check_chat(client, model))