Repository navigation
Sync master with upstream (2026-11-20) - #235
Merged
Merged
Conversation
…ml-org#32011) * fix(hexagon): make host build work with clang-cl on Windows ARM64 Two CMake fixes for building ggml-hexagon natively on Windows-on-ARM64 with x64-hosted clang-cl (validated on Snapdragon X Elite, discussion in qualcomm/GenieX#1390): - Some clang drivers (e.g. x64-hosted toolchains used on ARM64) don't predefine the arch macro, which makes winnt.h fail with "No Target Architecture". Define _M_ARM64/_ARM64_ explicitly for the host TUs. - The Hexagon SDK's incs/stddef include (AEEStdDef.h and friends) poisons the MSVC header chain when it leaks onto dependent TUs' include paths via htp_iface's PUBLIC includes. Keep it PRIVATE to the interface stub and system-level for the backend's own TUs (which include <AEEStdErr.h> directly). Originally validated 2026-08-26 as reported in the issue; applies cleanly to 15759e0. * hex-cmake: trim down excessive comments --------- Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
…gml-org#32166) The same-origin tests over the LAN address block forever on the Metal runners: the request to the host's own LAN address never returns and has no timeout, so the job runs until the 6h limit. get_lan_ip() now probes /health on that address with a short timeout and returns None when the host cannot reach it, which skips the test. The test requests themselves use the default request timeout so a hang fails the test instead of stalling the job.
…g#32167) The vision tests download their images from huggingface.co, which resolves to a private address on the runners inside the HF network. The media URL check refuses it with "host is not a public address" and the server-cuda job fails. The vision fixture enables media_allow_private_urls. The private address check keeps its own coverage in test_vision_image_url_private_address.
… release] (ggml-org#32168) * server: read a small temp file in the read_file tool test test_tools_builtin_read_file read its own file, which grows with every added test. On Windows the CRLF checkout puts it at 16664 bytes, over the 16384 byte read limit, and the tool refuses it. The test now writes a small file of its own and checks the exact content read back. * server: use a temp cwd in the x-tool-cwd tool test test_tools_builtin_cwd_header read test_tools_builtin.py through the x-tool-cwd header, which hits the same 16384 byte read limit on the Windows CRLF checkout, and wrote its shell marker into the source tree. The test now uses a temp directory as cwd, reads a small file from it and checks the shell marker lands there.
…ml-org#32183) The D=256 vec flash attention kernel with F8_E4M3 K and V uses 257 VGPRs (255 + 2 spilled) on gfx908, one over the limit of the check. The spill is a perf issue only, like the Q4_0 D=256 vec kernel already in the ignore list. Both variants of the kernel join the ignore list. [no ci]
* examples : clean-up old/obsolete stuff * examples : update references to removed files Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB) [no ci]
* ggml : bump version to 1.0.0 * examples : bump test-cmake (ggml/0)
[no ci]
* llama.cpp : bump version to 1.0.0 * readme : bump major version in link [no ci] * examples : bump test-cmake version [no ci]
This commit removes the llama version test-cmake and instead uses the version that is supplied by CMAKE_PREFIX_PATH. The motivation for this is to avoid issues when bumping major versions.
…ggml-org#32179) The kernel launches one block per row and 256-wide tile, so the {2, 16777217} test case asks for 16777217 blocks of 256 threads. HIP refuses a grid dimension above 2^32 threads and the ROCm jobs fail with "invalid configuration argument". The host now launches the grid in chunks of at most UINT32_MAX / 256 blocks and passes the chunk offset to the kernel. Real sizes fit in a single chunk and keep their timings.
* grammar : limit the number of alternatives * grammar : introduce an AST for grammar parsing * grammar : revert AST refactor and alternatives limit This reverts commits: - 84f3b57 - 393a707 * grammar: optimize big grammar matching by using left-factorization * grammar : preserve character class boundaries during left factoring * grammar : prevent infinite expansion during left factoring * grammar : use ASCII characters in comments * grammar : test empty suffixes after left factoring * grammar : deduplicate left factoring logic * grammar : add comments * grammar : use number parameters and clarify RSS output * grammar : clarify character class test names * grammar : remove performance benchmark * grammar : share immutable rules --------- Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com>
* ggml-vulkan: add optional device keepalive * simplify implementation, move ownership to backend, change env var to seconds --------- Co-authored-by: Acmmi <acmmi@Acmmis-MacBook-Air.local> Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
…sampling (ggml-org#31294) * spec : skip p_min for sampled MTP and DFlash2 drafts Under rejection sampling the acceptance at a position is 1 - TV(p, q), which the draft's top probability does not tell, so the p_min cutoff dropped draft tokens the target would have accepted. Skip it when MTP or DFlash2 samples the draft. draft-simple keeps it: each extra draft token costs a full draft model pass. * spec : replace the sampled-draft p_min skip with a warning p_min behaves as on master again for every drafter and sampling mode. Instead, log a warning when --spec-draft-p-min > 0 is used with --spec-draft-sampling probabilistic, suggesting p_min 0: under rejection sampling the draft's top probability does not tell how much of the draft the target accepts. The cut is not skipped because with MTP at long drafts it can still pay off. * spec : warn about p_min only for drafters that use it The n-gram drafters ignore p_min, and DSpark uses it as the cutoff of its confidence head, so the warning is skipped when only these are configured. --------- Co-authored-by: Pranesh Gonegandla <pgonegandla@nvidia.com>
* vulkan: add TOP_K_MASK Assisted-by: Claude Opus 5.5 * vulkan: fix the TOP_K_MASK barrier and use the common offset helper Assisted-by: Claude Opus 5.5 * vulkan: declare ggml_vk_top_k_mask in the common header Assisted-by: Claude Opus 5.5
* ggml : add BF16 support to RMS_NORM (CPU, CUDA) * cuda : use one helper for the F32/BF16 RMS_NORM launches
Signed-off-by: ynankani <ynankani@nvidia.com>
…l-org#32144) * hex-bufs: combine queue and fence buffers * hex-bufs: use n_threads for queue sizing * hex-mmap: optimize buffer stacking at the host to avoid iova fragmentation * hex-mmap: add simply retry for spurios mmap failures * hex-mmap: use eager mode for extended mmap retries
* requirements : support Python 3.15 torch 2.11.0, numpy 2.4, matplotlib 3.10, pandas 2.3, pillow 11.3 and cffi 1.16 have no Python 3.15 wheels, so the CI runner on 3.15 cannot install the requirements. Python 3.15 gets the first releases that ship 3.15 wheels through version markers, and every version below 3.15 stays as is. pyproject.toml now allows 3.15 and picks torch 2.15.0 there. * requirements : apply review suggestions numpy 2.5.4 from Python 3.14, matplotlib 3.11.2 and pandas 3.0.6 from 3.11, cffi 2.1.1 and pillow 12.3.0 for every version. cffi 1.16.0 has no 3.14 wheel and fails to build there, so this also fixes the tokenizer test requirements on 3.14.
…-org#32148) * ci: reduce qdc timeouts to avoid stalling pipelines and runners * ci: fix flake8 warnings
* hexagon: add SSM_SCAN Op support * hex-ssm-scan: fix multi-device spliting * hex-ssm-scan: remove redundant checks and make kernel use host config * hex-ssm-scan: remove direct ddr access * hex-ssm-scan: remove DMA ordering * hex-ops: update supported ops * hex-ssm-scan: rewrite the dma pipeline to match other ops (dbl.buf, etc) * hex-ssm-scan: remove all fallbacks, only dma/vtcm paths * hex-ssm-scan: add kernel_type to kparams and remove redundant recomputes * hex-ssm-scan: avoid stalls in the overall pipeline (dma flush between chunks) * hex-ssm-scan: vectorize softplus * hex-ssm-scan: optimize hvx loops * hex-ssm-scan: add HMX support * hex-ssm-scan: add support for GGML_HEXAGON_SSM_SELECT * hex-ssm-scan: cleanup kernel names * hex-ssm-scan: simplify mdev checks * hex-mdev: improve mdev row-split solver --------- Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* ggml-hexagon: IQ2_XXS and NVFP4 tiled repack layouts * ggml-hexagon: IQ2_XXS HVX matmul kernel * ggml-hexagon: NVFP4 HVX matmul kernel * ggml-hexagon: NVFP4 per-tensor scale * ggml-hexagon: HMX paths for IQ2_XXS and NVFP4 * ggml-hexagon: supports_op coverage for NVFP4 and IQ2_XXS * fix HMX IQ2_XXS dequant scale underflow in fp16 * hexagon: IQ2_XXS 2-gather tile decode with compact grid, small-weight tests * IQ2_XXS vpopcount sign parity and gather pipelining * pipeline IQ2_XXS HMX dequant gathers and add large_w tests * hex-iq2-xxs: rebase and cleanup Remove unused helper functions. Add lut_size to the overall vtcm required size. Trim down (way way down) the new tests. * hex-iq2-xxs: actually trim down the tests --------- Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
…l-org#32016) * dflash : move the DSpark markov head and backbone graphs to dspark.cpp Move-only: build_dspark_markov_head, graph_dsv4, graph_dsv41 and graph_mistral_large4 move verbatim. build_inp_target_features and build_dspark_markov_head become static members of llama_model_dflash so both files can use them. * dflash : detect the head and the backbone once, split the loaders per file load_arch_hparams sets llama_model_dflash::variant (DFlash, DFlash2, DSpark, LiliCorr) and ::backbone (base, DSV4, DSV4.1, Mistral Large 4). The head, backbone and base layer loaders become member helpers in dflash.cpp, dflash2.cpp, lilicorr.cpp and dspark.cpp. build_arch_graph dispatches on the backbone. The DFlash2 and LiliCorr loaders create the DFlash2 conv tensors through a helper in dflash2.cpp, right after their head tensors instead of in the base layer loop; the other tensors keep their creation order. The DFlash2 loader drops its copy of the conv group size check, which master never reached. * dflash : share the injection input, token embeddings and lm head lookups build_inject_inp (target features + fc + encoder norm), get_tok_embd and get_lm_head replace the copies in the base, DSV4, DSV4.1 and Mistral Large 4 graphs. build_inp_target_features is file-static again. * dflash : split the DFlash decoder into a graph per head graph_base creates the shared inputs and provides build_inject, build_inp_tokens, build_layers and build_logits. graph_dflash (dflash.cpp), graph_dflash2 (dflash2.cpp), graph_dspark (dspark.cpp) and graph_lilicorr (lilicorr.cpp) assemble them and add their head; the selector, scorer and markov head builders move to the file of their graph, and the DFlash2 local conv run by build_layers to dflash2.cpp. build_arch_graph dispatches on the backbone, then on the head. The op order and the tensor names are unchanged. * speculative : split the DFlash driver into a class per head common_speculative_impl_draft_dflash stays the DFlash implementation and becomes the base of draft_dspark, draft_dflash2 and draft_lilicorr. Its protected constructor only reads the draft metadata; init() then sets up the contexts with the options of the draft (draft length, backend sampling, device features, masked h_nextn). Each derived class overrides draft() with the path of its head from the former draft(). DFlash2 also overrides begin() to reseed its generator per request, and LiliCorr overrides process() to store the context features of each injected chunk, which its noise block carries as its state. The factory keeps the current routing: draft-dspark picks DSpark, draft-dflash picks DFlash2 or LiliCorr from the draft metadata and DFlash otherwise. draft-dspark with a DFlash2 or LiliCorr draft now runs the DSpark reader instead of a mix of both. sample_from_anchor is logged by DSpark only. * speculative : add the draft-dflash2 and draft-lilicorr types draft-dflash2 and draft-lilicorr name the DFlash2 and LiliCorr drafts, which draft-dflash keeps running too: it picks them from the draft metadata. draft-dflash2 and draft-lilicorr must match the draft model, draft-dspark rejects a DFlash2 or LiliCorr draft, a model that is not a DFlash draft is rejected, and a run uses at most one DFlash family type; these errors leave speculative decoding uninitialized. The type detected from the draft GGUF is now the concrete one, and draft-dflash2 and draft-lilicorr download the dflash sidecar. * docs : describe the DFlash2 and LiliCorr drafts Add the DFlash2 and LiliCorr sections and the new types, and explain that draft-dflash also runs DFlash2 and LiliCorr drafts. * speculative : drop the unused DFlash is_mrope Nothing reads it since the migration to common_batch (ggml-org#29385), which picks the number of M-RoPE position rows from the rope type of the draft. * doc: update spec dec * address review comments
* ui: move the reasoning effort into the model picker The chat form's row no longer holds the reasoning button: the model picker ends with a Reasoning row, in place of Manage models, that shows the level in use and opens the levels under it. A phone's model drawer has the same row. A desktop server with a single model, whose selector opens the manager instead of a picker, keeps the button beside it. Assisted-by: Claude (Opus 5.5 Medium) * ui: show a reasoning effort that is not the default on the model selector Next to the model on the closed selector, a set level shows as its word, and reasoning that is off as a small crossed-out lightbulb. At the default, which leaves it to the model, nothing more shows. Assisted-by: Claude (Opus 5.5 Medium) * ui: keep the model picker's footer in place while it opens in Safari WebKit pins a bottom-sticky element against the scaled box of a parent that zooms in, so the Reasoning row rode up over the model list until the opening animation ended. The desktop picker now scrolls its search and list in a box of their own, with the footer under it. Assisted-by: Claude (Opus 5.5 Medium)
…2648) graph_max_nodes() picked the nodes-per-tensor factor from lists of architectures. Each model class now gives it through a new virtual, llama_model::graph_nodes_per_tensor(), which defaults to 8. The architectures that need 32 or 64 override it in models.h, and DFlash keeps its hparams-based rule in dflash.cpp. LoRA adapter nodes are now added for every architecture, not only for the ones on the default factor.
The model modality icons match the ones the Llama apps use: glasses for vision, headphones for audio. The attach menu's list of what the model takes reads the same icons instead of its own microphone and picture. Assisted-by: Claude (Opus 5.5 Medium)
…org#32615) * ui: drop component tokens that only alias a semantic token A component token now exists only for what a component decides itself: a value no semantic token has, or one that only this component changes between themes. Components that take a semantic token as it is read it directly, so 88 of the 118 component tokens go, with no change to what is drawn. Also drops semantic tokens that repeat another one (status-* is fill-*, elevation-tooltip is elevation-surface), the unused info lamp tokens and 14 primitives nothing reads, and adds fg-soft for the label of an option that is not the picked one (tabs, toggle group). Assisted-by: Claude (Opus 5.5 Medium) * ui: add a short summary on top of DESIGN.md The rules most changes need, in one screen: components, colour, type, shape, sizes, surfaces, tokens and no arbitrary values. The sections below keep the detail. Assisted-by: Claude (Opus 5.5 Medium) * ui: derive design tokens by rule and drop the alias layers Semantic tokens are either a pick (a primitive per theme) or a derivation, mixed from another semantic token by a fixed rule and declared once: hover fills, status surfaces and lines, dark shadows (--elevation-strength), the middle of a matte edge, a lamp's rim and glyph. Primitives keep colours only. shadcn's colour names stay as Tailwind utilities and no longer exist as variables; neutral semantic utilities that duplicated them are gone. Assisted-by: Claude (Opus 5.5 Medium) * ui: set all text through type roles and move off-scale values onto the scale Tailwind's text sizes and bracketed sizes become the type roles, with a new text-caption-sm (10px) for tags and counts. Body and caption roles no longer set a weight, and line heights are written as Tailwind writes its own, so the swap from text-sm and text-xs changes no pixel. A unit test keeps old sizes from coming back. Arbitrary values with a scale equivalent use it (z-20, max-w-4/5, min-w-32), and the two easing curves repeated across the app are ease-out-quint and ease-sheet. DESIGN.md is updated and regenerated. Assisted-by: Claude (Opus 5.5 Medium) * ui: bump @ggml-org/llama-ui to 0.5.0 Removed and renamed tokens are breaking for an app that reads them. Assisted-by: Claude (Opus 5.5 Medium) * ui: add a display type role for the biggest headline The greeting on a new chat keeps its 30px from md up, now as text-display instead of a Tailwind size. Assisted-by: Claude (Opus 5.5 Medium) * ui: one corner scale, one set of shadows and three more derived tokens Six corner roles named values Tailwind's rounded-md to rounded-4xl already have, and components used the Tailwind names nine times out of ten. The roles are gone; the whole scale is computed from --radius. Tailwind's shadow sizes were restated next to the elevation tokens and did not follow the theme, so they were five times fainter in dark. The five uses take elevation tokens and a unit test keeps both from coming back. A button group's seam is the button's own text colour at 25%, the dark status text is the dark status fill, and the pressed shadow scales with --elevation-strength like the inset one. Assisted-by: Claude (Opus 5.5 Medium)
…gml-org#32653) * llama : move the indexer key rotation choice into the model classes The KV cache forced a Hadamard rotation for the lightning indexer keys based on a list of architectures. The virtual llama_model::indexer_k_rot() now returns true for the same models. DeepSeek4.1 inherits from DeepSeek4 but was not in the list, so it overrides it back to false. * llama : rename indexer_k_rot() to lid_k_hdmd()
* model: load `ffn_gate_up_exps` scales The generic scale pass loads per-expert scales for ffn_gate_exps, ffn_up_exps and ffn_down_exps but not for the merged ffn_gate_up_exps. Load ~ffn_gate_up_exps_s there~ and expose it as ffn_up_exps_s ~when the layer has no separate up tensor,~ since build_moe_ffn applies up_exps_s to the merged gate_up output. Inkling 2 now passes the expert scales to build_moe_ffn like the other MoE models. * Update src/llama-model.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Update src/llama-model.h Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
…2570) A printable key pressed on the chat screen while no field, editor, dialog or menu has focus now moves focus to the chat form, and the character lands there. Assisted-by: Claude (Opus 5.5 Medium)
* ui: load highlight.js on first use highlight.js and every lowlight language (~900 KB raw) moved out of Llama Chat's first load into a chunk that loads with the first code block. - @ggml-org/llama-ui/code: highlightCode stays synchronous. Until highlight.js has loaded it returns the code escaped as plain text and starts the load; a $derived or $effect that called it runs again once it has loaded. New loadHighlighter() and isHighlighterLoaded(). - Markdown: rehype-highlight runs through a lazy rehype step, like KaTeX's: the first message with a code block waits for it, every other one renders in the same pass. First-load JS of the chat page: 3.47 -> 2.56 MB raw, 1.05 -> 0.75 MB gzip, 0.84 -> 0.60 MB brotli. Assisted-by: Claude (Opus 5.5 Medium) * ui: load each highlight.js language on first use highlight.js now loads one language at a time instead of all ~190 at once with the first code block: every language is its own chunk, and the core loads with the first one. - @ggml-org/llama-ui/code: highlightCode looks a language or alias up in a generated table (scripts/generate-highlight-languages.mjs), loads it with the languages it embeds (CSS and JavaScript inside HTML, say) and returns plain text until then; a $derived or $effect that called it runs again once it has loaded. Auto-detection picks among lowlight's 37 common languages, loaded the first time something auto-detects. New loadHighlightLanguages(), isHighlightLanguageLoaded(), resolveHighlightLanguage() and getLoadedHighlightLanguages() replace loadHighlighter() and isHighlighterLoaded(). - Markdown: rehypeLazyHighlight highlights with lowlight's core and the languages a message uses, waiting only for one it is the first to use. rehype-highlight is no longer in the pipeline, since it bundles lowlight's common languages. First-load JS of the chat page stays at 2.58 MB raw, 0.76 MB gzip, 0.61 MB brotli: no highlight.js code, only the 14 KB language table. The first code block now loads its language (1-12 KB for most) and the 20 KB core instead of 900 KB. Assisted-by: Claude (Opus 5.5 Medium) * ui: name the code-block class names and shell language as constants Assisted-by: Claude (Opus 5.5 Medium)
* fix(server-bench): prevent float bin-count TypeError in histogram generation Symptom: benchmark crashes while generating gen_rate histogram. Root cause: np.ceil(...) returns numpy.float64, and that value was used where an integer count/shape argument is required. Fix: cast ceil result to int before +1, ensuring bin count is an integer. Result: histogram path completes and gen_rate.png is generated reliably. * Update scripts/server-bench.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
llama-tts built the mtmd context from mtmd_context_params_default(), which hardcodes 4 threads, and never copied n_threads over, so -t never reach the mtmd side. Copy params.cpuparams.n_threads into mtmd_params.n_threads, as mtmd-cli and the server do; with no -t this now follows the resolved default. Verified: -t 1 now runs single-threaded, -t 4 is unchanged, and the produced WAV is identical. Assisted-by: Claude
The unscoped llama-ui on npm is a 0.0.1 placeholder. After the scoped publish, the workflow renames the same build and publishes it again at the same version, with a readme note pointing at the scoped package. Assisted-by: Claude (Opus 5.5 Medium)
…ggml-org#32670) * ui: search message content and Discover models from the search dialog The search dialog matched chat titles only. It now also scans what was said in each chat (content, reasoning, tool calls and results, attachment names and text), newest chat first, stopping at the first match per chat and after 20 chats, and shows a snippet around the match. Ids, timestamps and base64 payloads are not searched. With Discover on, it also lists matching catalog models and, once typing settles, Hugging Face repos; picking one opens Discover on that repo. Assisted-by: Claude (Opus 5.5 Medium) * ui: keep the search dialog one width whatever it lists The shared dialog sizes itself to its content from md up, so the search was narrow while empty and widened as results came in. It now takes the full max-w-2xl there, less a margin on narrower windows. Assisted-by: Claude (Opus 5.5 Medium)
…gml-org#32531) * ui: add a voice engine for capture, peaks and transcription Voice input needs a live waveform, a clip that can be sent and drawn, and a transcript, whether the model streams text while the user speaks or not. This adds the engine for it in src/lib/voice, with no imports of app stores or components, so it can move into a package later. - VoiceCapture: one microphone start gives live levels (RMS on a dB curve, fixed history, newest last), PCM16 mono chunks at 16 kHz from an inline AudioWorklet (ScriptProcessor fallback), and on stop a 16-bit mono WAV clip with its duration and peaks. Tracks, nodes and the context are released on stop, cancel, a failed start and page hide. - computePeaks / computePeaksFromAudio: normalised peaks of samples, a buffer, or a stored base64 or Blob file. - transcribeClip: POST /v1/audio/transcriptions, with the server's own error message on failure. - RealtimeTranscriber: the /v1/realtime?intent=transcription socket, queueing audio until session.created, a final commit, and a typed RealtimeUnsupportedError when the model cannot stream or the socket is refused (a browser websocket cannot send the API key header). - startVoiceSession and VoiceRecorder: try realtime, fall back to one HTTP transcription of the clip, and expose levels and live text. Assisted-by: Claude (Opus 5.5 Medium) * ui: add voice waveform and transcript components to Llama UI Voice input needs three visuals that the app can feed with its own recording and transcription logic: the live waveform while recording, a playable waveform of a recorded clip, and a clip with its transcript that opens on hover, keyboard focus or a tap. VoiceWaveformLive draws input levels as many bars as fit the width, newest on the right, and holds still with reduced motion. VoiceWaveform plays a clip through its own audio element, fills the played part in the primary colour, seeks by press, drag and arrow keys (a slider) and pauses any other clip that was playing. VoiceTranscriptReveal keeps the transcript in the DOM for screen readers while it is closed, and shows transcribing and error states. The waveform math (resampling peaks, bar capacity, seek and time formatting) is in src/lib/voice with unit tests. The components are exported from the package root and from @ggml-org/llama-ui/voice, which Llama Chat resolves to the sources like the other entry points. Assisted-by: Claude (Opus 5.5 Medium) * ui: add stories and tests for the voice components Storybook gets UI/Blocks/Voice: the live waveform fed by a fake microphone meter in a chat form row, the clip waveform in its sizes and states, and the transcript reveal in each state and on touch. The demo audio is a WAV synthesised in the story, so no binary file is added. The play functions check seeking by keys and by press, the reveal on hover, keyboard focus and a tap, and a clip without audio. Pressing play needs a real click before a browser plays audio, so a client test covers play and pause and one clip pausing another. Assisted-by: Claude (Opus 5.5 Medium) * ui: store voice messages with their transcript and send it to models that cannot hear A voice message recorded in the chat form is an audio attachment like an uploaded file, plus what the app needs to draw and read it. The audio attachment gets four optional fields: voice (recorded in the app), transcript, peaks and durationMs. Stored rows, export and import carry them as they are, so no migration is needed and older messages keep working. The transcript is for reading. A model with audio input still gets the audio alone; a model without it (the user switched models later) gets the transcript as a labelled text part in place of the audio. When the model's modalities are not known yet, or there is no transcript, the audio goes as before. A transcript often comes back after the message was sent. The new voiceTranscriptsStore follows each pending transcript by the clip's name, hands it to a send that happens after it arrived, and otherwise writes it into the stored attachment of the message it went out with, in the open chat or, if the user left it, in the database. Copying a message now includes its voice transcript. Assisted-by: Claude (Opus 5.5 Medium) * ui: add Voice settings and resolve what the microphone does per model The user should not have to think about which kind of voice input is in use, so the app decides it from the conversation's model and two settings. resolveVoiceMode is a small pure function: with Voice input on Automatic and a chat model that has audio input, a recording is a voice message, transcribed by that same model so a router swaps nothing. Otherwise it is dictation, with the chat model when it can hear, else, in router mode, with the chosen transcription model. When no model can transcribe, the microphone is unavailable and says why. Settings get a Voice section of their own rather than more rows in General, which is already long: these four belong together and one of them lists the server's models. - Voice input: Automatic (default) or Always dictate. - Transcription model: Same as chat model (default) or, in router mode, any model with audio input; on a single-model server it only says whether that model can transcribe. It notes that dictating with another model loads it beside the chat model. - Transcribe voice messages: on by default. - Transcription language: empty for automatic. The keys live in the settings registry like the others, so defaults, storage, reset and export need nothing new. The model picker is drawn by SettingsChatVoiceTab, the way Extensions and Tools draw their sections. Assisted-by: Claude (Opus 5.5 Medium) Co-authored-by: Nicolas Mowen <14866235+NickM-27@users.noreply.github.com> * ui: record voice messages and dictation from the chat form The microphone now does what the conversation's model allows, without the user choosing a mode: its label says "Send a voice message" or "Dictate", and when no model can take audio it is disabled and says why (shown in router mode only when some model there has audio input, so the reason can be acted on; a single-model server without audio keeps the send button as before). The microphone errors keep their warning state and toast, now also for the voice engine's own capture errors. Recording goes through the voice engine's VoiceRecorder instead of the MediaRecorder flow. While it records, the actions row becomes a recording row: cancel, the live waveform, the elapsed time and stop. Escape or cancel drops the recording. The form keeps its two rows while recording, so the message stays visible on desktop and the row fits the phone layouts and the expanded sheet. - Dictation writes the live transcript into the message at the caret as it streams. DictationSpan replaces only the dictated stretch, moves it when the user types before it, keeps what is typed after it, and hands text the user edits over to them. When nothing streams, the row shows Transcribing... after stop and the text goes in once it is back. Cancel takes the dictated text out again. No audio is attached. - A voice message streams its live text, if any, under the waveform, not into the message. On stop the clip is attached as a voice file with its peaks and length, and its transcript is followed by the voice transcripts store (unless Transcribe voice messages is off). Realtime transcription is skipped for servers reached through a tunnel or with an API key, which a browser websocket cannot carry. The first dictation with a separate router model shows a one-time note that it loads beside the chat model. The form takes an optional voiceRecorder, so tests and stories can run it without a microphone. Assisted-by: Claude (Opus 5.5 Medium) * ui: draw audio attachments as playable waveforms with their transcript A voice message should read as a clip, not as a file chip. Audio attachments now render with Llama UI's VoiceWaveform inside VoiceTranscriptReveal, through a direct branch in the core attachment item (voice is core, so it does not go through the extension renderers): the removable sm chip in the chat form, and the md clip in a sent message, on the user bubble's surface. - The audio plays from an object URL made from the stored base64 (or the uploaded file's data URL), revoked when the clip leaves the screen. - A clip stored without peaks, an older voice message or an uploaded audio file, gets them computed from its audio once per session, in memory only; uploaded files keep their name under the waveform. - The transcript opens under the clip on hover, keyboard focus or a tap, and shows Transcribing... while the voice transcripts store waits for it, Transcription unavailable when it failed, and nothing when none was asked for. - The full-screen preview keeps its player and shows the transcript. Tests run the chat form with a fake recorder: the microphone per mode, the recording row, dictation around typed text, the Transcribing... state, cancel, and a voice message chip whose transcript arrives later and goes with the send. Stories show the form recording each kind, a recorded chip, and sent voice messages with the transcript, pending, and computed peaks. Assisted-by: Claude (Opus 5.5 Medium) * ui: drop the old audio recorder and proxy realtime transcription in dev The voice engine replaced AudioRecorder, so it goes. The dev server now proxies WebSockets on /v1, so realtime transcription works under vite dev. The VoiceCapture test waits for the tone to be recorded instead of a fixed 700 ms, which a loaded test run could miss. Assisted-by: Claude (Opus 5.5 Medium) * ui: dictate with a model that is already loaded when the chat model cannot hear The Transcription model setting's Automatic choice now falls back to a router model with audio input that is already loaded, so dictating with a text-only chat model works without picking a model and loads nothing. A picked model still wins, and the empty value is a named constant. Idea from upstream PR 27567, by NickM-27. Assisted-by: Claude (Opus 5.5 Medium) Co-authored-by: Nicolas Mowen <14866235+NickM-27@users.noreply.github.com> * ui: fill the recording waveform, and open voice message transcripts - The live waveform fills the row from the start: a recording shorter than the row is stretched over all its bars, a longer one scrolls with the newest level on the right. No dotted placeholder. - A voice message's transcript is open under its clip, in the chat form while it comes in and in the sent message, with the toggle beside it to fold it. - Settings > Voice > Collapse transcripts (off by default) keeps them folded instead, opening on hover, keyboard focus or a tap. Assisted-by: Claude (Opus 5.5 Medium) (cherry picked from commit 2a93f2f) * ui: record voice without ducking other audio, from the first word - Voice input asks for the raw microphone signal: no echo cancellation, noise suppression or auto gain. Echo cancellation made the system treat a recording as a call, which lowers or interrupts other playback, and auto gain faded the first words in. - One AudioContext serves every recording, with the capture worklet loaded once, suspended in between and, where the browser allows it, without an output device, so recording never touches the speakers. - The chat form prepares it when it gets focus (no microphone, so no recording indicator), and a mouse starts recording on press. - A recording starts counting once audio flows: until then the bar says Starting... instead of running the timer. Development builds log how long each step of a start took. - Settings > Voice > Microphone: Automatic (the system default, or the built-in microphone when the default is a Bluetooth headset, which would switch to its low-quality call mode), System default, or a listed microphone. Assisted-by: Claude (Opus 5.5 Medium) (cherry picked from commit 76643e7) * ui: show voice text live in the message box, and follow transcripts as spoken - A model that hears but has no live ASR (e.g. LFM3-Omni) now gives text while you record: the realtime client commits the audio piece by piece, at the pauses the levels show, one piece at a time, and the final commit takes only what is left. - A voice message's live text shows as the empty message box's placeholder. Clicking into the box makes it text to edit there, and what is said next follows it. On stop the text leaves the box and goes with the clip; when it was edited, the edit is the clip's transcript. - Llama UI gains VoiceSpokenText and spokenLength(): spoken words in the full foreground colour, which they keep, the rest muted. A clip's transcript follows its playback (estimated over the parts with speech, as there are no word timings), and live text brightens as it arrives. - The recording bar has a send button that stops and sends in one go. Assisted-by: Claude (Opus 5.5 Medium) (cherry picked from commit ce77301) * ui: let the capture test start with silence A capture starts with its first audio block, which can come before a stream's first sound, so the steady tone is checked from where it begins. Assisted-by: Claude (Opus 5.5 Medium) (cherry picked from commit a41ee53) * ui: show live voice text sooner, and keep silence from becoming "yeah" - A model that transcribes per commit (LFM3-Omni) gets shorter pieces: cut at a 0.35 s pause after speech, or at the next quiet moment once a piece reaches 3 s, so the text follows the speaker within a few seconds in dictation, the message box and voice mode. - When the recording stops, the audio after the last piece is committed only if speech was heard in it. LFM3-Omni transcribes a commit of pure silence as "Yeah." (or "はい。"), which ended up at the end of the text. A recording whose only piece is that rest still commits it. Assisted-by: Claude (Opus 5.5 Medium) (cherry picked from commit 43e52cb) * ui: transcribe a recording cut at pauses once more as a whole Live text comes piece by piece, cut at the speaker's short pauses, and a model reads each piece as a sentence of its own: "I went. To the store." When the recording ends, one pass over the whole clip gives the final text, punctuated by meaning, in dictation and in voice mode alike. The pieces' text stays the live text until then, and is kept when that pass fails or the recording is longer than two minutes. Assisted-by: Claude (Opus 5.5 Medium) (cherry picked from commit 359dd96) * ui: give live text over HTTP for models without a realtime session A model that hears but cannot hold a realtime session, such as Qwen3-ASR, was transcribed only after the recording, so dictation showed no text while speaking. The recording is now cut at the speaker's pauses as with a realtime session, and each piece sent to /v1/audio/transcriptions on its own, one at a time, its text joined live. The audio is kept from the start, so what is said while the socket answers goes with the first piece. The whole clip is still transcribed once more at the end, and the pieces' text is kept if that fails. A server a websocket cannot reach (an API key, a remote server) gets live text the same way. Assisted-by: Claude (Opus 5.5 Medium) * ui: say when the browser processes the microphone despite the raw request The browser keeps one processing setting per microphone: while a call, another tab or voice mode has it open with echo cancellation, a dictation that asked for the raw signal is processed too, sounds distant, and the system may lower other sound. The capture now warns when it gets processing it did not ask for, and its development log names the microphone, its rate and what processing is on. Assisted-by: Claude (Opus 5.5 Medium) * ui: keep the microphone beside the stop and send buttons While a reply streams, or once the message has text, the microphone no longer gives way to the main button: it stays beside it as a quieter, secondary button, so more can be dictated without clearing the message or waiting for the reply. Assisted-by: Claude (Opus 5.5 Medium) * ui: show the voice components' stories with Llama UI VoiceWaveform and VoiceTranscriptReveal are Llama UI components, so their stories move to the library's Storybook, under Llama UI/Voice, with the demo audio they play. Llama Chat's Storybook shows them there too, and its own voice stories take the demo audio from the library. VoiceWaveformLive stays with Llama Chat, as its demo records through the chat form's microphone. Assisted-by: Claude (Opus 5.5 Medium) * ui: add a noise suppression setting for voice input Recordings stay raw by default. Settings > Voice can now switch the browser's noise suppression on; echo cancellation and automatic gain stay off either way, as they interrupt other sound and fade the first words in. Assisted-by: Claude (Opus 5.5 Medium) * ui: keep live voice text coming when the room is never quiet Pieces were cut only at a quiet moment, so with music or talk in the room none was ever cut and no text showed until the recording ended. A piece half as long again as its longest length is now cut where it is. Assisted-by: Claude (Opus 5.5 Medium) * ui: add echo cancellation and automatic gain settings for voice input Settings > Voice now has a switch for each kind of microphone processing the browser offers. All three stay off by default, so recordings are raw; each one says what switching it on costs. Assisted-by: Claude (Opus 5.5 Medium) * ui: put the microphone after the stop button, filled like it While a reply streams, the microphone is the last button of the row and looks like the stop and skip buttons beside it, instead of a bare icon before them. Assisted-by: Claude (Opus 5.5 Medium) * ui: say when dictated text is being refined, and send it refined After a recording with live text, the whole clip is transcribed once more so pauses do not end sentences. The form now says so while that pass is out, and a send asked for meanwhile waits for it, so the message goes out with the refined text rather than the pieces. Assisted-by: Claude (Opus 5.5 Medium) * ui: keep the send button in place and put the microphone beside it The send button no longer gives way to the microphone on an empty message. The microphone stands before it as a quieter, tertiary button, with the same tooltip, warning and recording states as before. Assisted-by: Claude (Opus 5.5 Medium) * ui: let a recording be sent while a reply streams, once it has words While a reply streams, the recording row now shows its send button as soon as the first transcribed words are in the message, so the recording no longer has to be stopped first. The microphone also stays beside the send button once a voice message is attached. Assisted-by: Claude (Opus 5.5 Medium) * ui: raise quiet speech for transcription, and say when it is too quiet A quiet microphone gave the model near silence to transcribe. A new Voice setting, Boost quiet speech (on by default), raises speech to a level speech models hear well, up to eight times louder: the audio streamed for live text as it goes, and the whole recording in one pass when it stops, so its first words get the same gain as the rest. Only speech is measured, never the silence around it, loud speech is left as it is, and a soft limiter keeps raised peaks from clipping. When speech stays quiet even at the highest gain, the recording row says so. Assisted-by: Claude (Opus 5.5 Medium) * ui: keep the voice engine free of app code The engine in lib/voice imported the microphone messages from the app's utils and the WAV type from its enums, the only two things keeping it from moving into a package of its own. The microphone checks and messages move into the engine (voice/support.ts), next to the capture that needs them; the uploaded-file helpers stay in the app's utils. Assisted-by: Claude (Opus 5.5 Medium) * ui: start voice sessions from one voice service How a recording reaches the server (routes, API key, whether the realtime socket can be used) and how the Voice settings become capture and transcribe options were built inside the chat form's voice hook, and copied into every other feature that records. VoiceService builds them in one place: a feature asks it for its session options and adds only its own callbacks. Assisted-by: Claude (Opus 5.5 Medium) * ui: tell speech from the room in one place The pause detector, the speech gain and (in voice mode) the listener for speech over a reply each followed a noise floor with their own numbers. A SpeechDetector now does it for all of them, in dB, with one set of settings in VOICE_SPEECH. Assisted-by: Claude (Opus 5.5 Medium) * ui: let any text field take dictation Writing dictated text into a field (at the caret, around the user's typing, replaced by the whole recording's text when it comes) moves out of the chat form's hook into a plain createDictation(). The hook keeps what is the chat form's own: voice messages, toasts and the hint. Assisted-by: Claude (Opus 5.5 Medium) * ui: move the recording row into Llama UI VoiceRecordingBar draws the row shown while recording (cancel, waveform and time, send, and a caption or warning under it) from data alone, so voice mode or a notes page can show the same row. The chat form passes its labels and its microphone button as the stop button. Assisted-by: Claude (Opus 5.5 Medium) * ui: keep the voice engine's Svelte recorder behind its own entry VoiceRecorder, the one part of lib/voice written with runes, moves to lib/voice/svelte, so the rest of the engine is plain TypeScript any app can use without Svelte. Assisted-by: Claude (Opus 5.5 Medium) * ui: move voice input into its own package, Llama Voice The voice engine (capture, the speech signal, transcription over the realtime socket or HTTP) and dictation into a text field move from the app to ui/lib/voice as @ggml-org/llama-voice, so another app, such as a notes app, can use them. The Svelte recorder is its ./svelte entry. Llama Chat compiles the sources, as it does Llama UI's, and keeps its server and settings wiring. The package has its own type check, tests and build, and a CI job for them. Assisted-by: Claude (Opus 5.5 Medium) * ui: show which model does what in voice input The model picker and the Models manager mark each model that hears: Dictation on the one that transcribes dictation, Speech to text on transcription models such as Qwen3-ASR, and Voice on chat models that take voice messages. A loaded model kept only for dictation sits apart from the chat models, a model can be made the dictation model from its row, and Settings > Voice groups the transcription models apart from the chat models with audio input. A voice store now resolves what the microphone does, for the chat form and the model lists alike. Assisted-by: Claude (Opus 5.5 Medium) * ui: warn of a quiet microphone inside the recording row The too-quiet line under the waveform made the form grow and jump. The warning now stays inside the row: the waveform turns the warning colour, an icon sits by the time, and the advice opens above the row in a tooltip for a few seconds, then on hover or focus. Screen readers still hear it when it appears. Assisted-by: Claude (Opus 5.5 Medium) * ui: open the microphone while the pointer is on its button The first seconds of a recording could come out quiet while the microphone settled, and the too-quiet warning showed for them. With the pointer or the focus on the microphone button, the microphone now opens ahead of the press (only when the page may already use it, so nothing asks), and the recording takes it at once; it is let go shortly after the pointer leaves without a press. The warning also waits until the speech has stayed too quiet for two seconds. Assisted-by: Claude (Opus 5.5 Medium) * ui: tidy voice input after review Fix a microphone left open when a recording is cancelled while it starts or the pointer leaves while it opens, a capture that is still letting go silencing the next one, and live text stopping after a realtime error. Share the dictation model choice, the transcript lookup, base64 decoding, PCM joining and stopping a stream instead of repeating them, and export only Llama Voice's public API. Assisted-by: Claude (Opus 5.5 Medium) * ui: pin Llama Voice's dev dependencies to Llama UI's versions Assisted-by: Claude (Opus 5.5 Medium) * ui: let demos speak into a synthetic microphone Assisted-by: Claude (Opus 5.5 Medium) * ui: keep the microphone for voice input, off the audio files entry The attach menu's Audio Files entry used a microphone, which now sits beside the form's own microphone button. The microphone stands for recording and dictation only; a model's audio modality is headphones, and the audio files entry is a file with headphones. Assisted-by: Claude (Opus 5.5 Medium) * ui: set voice input on the slimmed design tokens Voice input's components use the type roles, the one corner scale and shadcn's utility names for the neutral colours, as the rest of Llama UI and Llama Chat do since the design tokens were slimmed down. Assisted-by: Claude (Opus 5.5 Medium) * ui: name voice states, platform strings, patterns and tunings Addresses the review on the voice PR: - Llama Voice: VoiceCaptureState, VoiceRecorderStatus, RealtimeState and RealtimeUnsupportedReason are enums (src/enums.ts, exported); DOMException names, AudioContext states, transcription form fields, socket protocols, device kinds, worklet messages, error messages, regexes and tuning values are named constants. - Llama UI: VoiceTranscriptStatus, VoiceWaveformSize and VoiceRecordingBarStatus are enums; KeyboardKey, MouseButton and PointerType join its UI enums; the voice components' sizes, labels, timings, selectors and patterns move to constants/voice.constants.ts. The voice unit tests move from src to tests/unit. - App: ChatFormVoiceState is an enum, the recording bar's status comes from a map, and the voice code uses the enums above, BooleanString, HEADERS and new voice constants in place of inline strings, regexes and numbers. Assisted-by: Claude (Opus 5.5 Medium) * ui: keep all of Llama UI's tests under tests The Model UI formatting spec moves to tests/unit and the button spec to tests/client, next to the voice specs; the test projects now read only tests/unit and tests/client. Assisted-by: Claude (Opus 5.5 Medium) * ui: show a transcript's words without the model's own format Qwen3-ASR names the language before its text (language French<asr_text>...) and VibeVoice-ASR answers with timed segments as JSON, which the server passes on as they are. The voice engine now takes both off, over HTTP and over the realtime socket, also while segments still stream, so the message box and the clip show the words alone. Assisted-by: Claude (Opus 5.5 Medium) * ui: dictate when a speech-to-text model is picked for the conversation A speech-to-text model has audio input, so on Automatic the microphone recorded a voice message for it, which such a model has nothing to answer with. It now dictates, as the model's badge already says it is for. Assisted-by: Claude (Opus 5.5 Medium) * ui: say why a live-only model cannot transcribe behind an API key A browser websocket cannot send the API key, so the realtime socket is left out on a server that needs one, and a model that only transcribes over it (Voxtral Realtime) answers the HTTP route with a 501. The clip and the dictation toast now say so instead of only that transcription is unavailable. Assisted-by: Claude (Opus 5.5 Medium) * ui: pick the dictation model with a microphone on its row The model that transcribes dictation was marked with a filled Dictation badge. It is now marked by a red microphone among the row's actions in the model picker, beside the heart, and every other model that hears shows the same microphone on hover: a click makes it the transcription model, and a click on the chosen one hands the choice back to Automatic. The badge keeps saying what kind of hearing model a row is, and the Models manager marks the dictation model with a small microphone beside its name. Assisted-by: Claude (Opus 5.5 Medium) --------- Co-authored-by: Nicolas Mowen <14866235+NickM-27@users.noreply.github.com>
* mtmd: drama3 voice cloning The Fish S2 codec encoder (fish_s2_spkenc) now also serves drama3: - conversion: drama3 writes the codec encoder and the audio encoder KV, shared with fish_s2 - fish_s2_spkenc: outputs the codes of the clip (gen-side reference) and, in a drama3_gen mmproj, builds the drama3 backbone rows (E_lm[semantic] + E_cb[1][code 1]) / sqrt(2) - drama3_gen tower: teacher forcing when the caller gives the codes of the next frame - drama3 pipeline: reference-bracket-v1 prompt with one reference (vLLM-Omni fish_omni), the reference frames go through the backbone as codec rows and through the tower with their codes - mtmd_gen_state_get_ref_codes(): read-only access to the reference codes of a gen state Claude-Session: https://claude.ai/code/session_01PpX3DmaxNSmJbxGaxnUBrX * fix correctness * clean up * rm redundant code * fish_s2: rm unused import --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* server: add more decision tests * fix ci * also test audio
* ci : disable musa.Dockerfile from docker.yml This commit disables musa in docker.yml due dl.mthreads.com is sometime unavailable. Refs: ggml-org#32672 (comment) * remove the SKIP_MUSA option
…org#32680) Llama Chat compiles @ggml-org/llama-voice from ui/lib/voice/src, but the CMake stage and the Docker UI stages copied only ui/lib/ui, so their npm build could not resolve the voice imports. Stage ui/lib/voice/src next to Llama UI and track it as an input of the CMake build. Assisted-by: Claude (Opus 5.5 Medium)
…gml-org#32675) Assisted-by: Claude (Opus 5.5 Medium)
Upstream: e46069c Preserve the fork README and GGML_HIP_UNSAFE_MATH default OFF. Assisted-by: Claude Fable 5.2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge upstream
ggml-org/llama.cpp:masterthroughe46069c40989(ui: fix the flaky reset check of the MermaidPreview Flowchart story (#32675)), bringing in 248 commits since the current fork master.Preserve the fork README and
GGML_HIP_UNSAFE_MATHdefault OFF. The README was the only merge conflict; it was resolved by retaining the fork version. The merged tree otherwise matches upstream except for the existing HIP unsafe-math option in two CMake files.Upstream changes in this range that touch the fork's HIP build files: the minimum ROCm/HIP version in
ggml/src/ggml-hip/CMakeLists.txtgoes from 6.1 to 6.3, and the ggml version goes from 0.29.0 to 1.0.0.Validation:
llama-cli,llama-server, and selected test targets passed.Pending before merge: HIP build, GPU smoke tests, and performance regression testing.
AI assistance: Claude Code prepared the merge, resolved the README conflict, ran validation, and created this draft PR.
🤖 Generated with Claude Code