Repository navigation
Description
Problem / Background
The reference Qwen3-VL processor renders a video as one <t.t seconds> text marker plus its own <|vision_start|>...<|vision_end|> block per temporal patch group, and splits each video_grid_thw row (t, h, w) into t rows of (1, h, w) for MRoPE. mlxcel renders one flat, timeless video block. Upstream found that without the markers the model reads a clip as a spatial collage (a moving square described as "several squares"); Neywa-Labs/mlx-vlm#2142 added the per-group rendering and Neywa-Labs/mlx-vlm#2413 took the timestamps from the frames actually sampled rather than an assumed 2 fps. Reference: _video_group_timestamps and _timestamped_video_placeholder in https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/qwen3_vl/processing_qwen3_vl.py, and the get_rope_index split in https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/qwen3_vl/language.py.
Current Behavior
insert_qwen_vl_media_tokens(src/multimodal/qwen_vl.rs:558-700) expands each video placeholder into one run ofgrid_t * (h/m) * (w/m)video tokens framed by a single start/end pair. It works on token ids and has no tokenizer.qwen_vl_mrope_positions_from_tokens(qwen_vl.rs:949-1073) treats each video run as one(t, h, w)grid and advancescurrent_posbymax(t, h/m, w/m).- The server video path (
src/server/model_worker.rs:3440-3492, also used bymlxcel generateon the engine) callsvideo::load_video_source_with_policy(src/multimodal/video.rs:1350-1384), which computesuniform_indices(meta.total_frames, nframes)andmeta.fpsbut returns onlyVec<DynamicImage>, so the timing is lost.Qwen2VLMediaInput::Video(src/vision/processors/qwen2_vl.rs:138-141) carries frames only. - The same processor and expansion serve Qwen2-VL and Qwen2.5-VL, whose reference processors do not emit timestamps.
Proposed Solution
- Timing. Add
pub struct SampledVideo { frames: Vec<DynamicImage>, frame_indices: Vec<usize>, source_fps: f64 }andload_video_source_sampled_with_policyinvideo.rs; makeload_video_source_with_policya thin wrapper returning.frames. The worker uses the new function and keeps the timing beside the frames. - Timestamps. Port
_video_group_timestamps: padframe_indicesby repeating the last index to a multiple oftemporal_patch_size(the processor's field,qwen2_vl.rs:32), frame time isindex / source_fps, group time is the mean of the group's first and last frame time. Error if the padded count differs fromgrid_t * temporal_patch_sizeorsource_fpsis not positive and finite. - Family flag. Add
video_timestamps: booltoQwenVlmPromptInfo(qwen_vl.rs:28-34), true for Qwen3-VL, Qwen3-VL-MoE, Qwen3.5-VLM and Qwen3.5-MoE-VLM, false for Qwen2-VL and Qwen2.5-VL (their output stays byte-identical). - Expansion.
compute_qwen_vl_media_embeddings(src/multimodal/vlm_runtime.rs:942) gainsvideo_timings: &[Vec<f64>](group seconds per video) andencode_text: Option<&dyn Fn(&str) -> anyhow::Result<Vec<i32>>>; the worker passes its tokenizer's encode without special tokens. Whenvideo_timestampsis set, each video placeholder (including its surrounding start/end pair) becomes, per groupg,encode_text(format!("<{:.1} seconds>", t_g)), start,(h/m)*(w/m)video tokens, end. A timestamp family withencode_text == Noneis an error, not a silent fallback. The already-expanded-prompt validation path must accept this shape. - MRoPE. For timestamp families, expand each video grid
(t, h, w)totgrids(1, h, w)beforeqwen_vl_mrope_positions_from_tokens, so each group run is positioned on its own and the timestamp text between runs gets text positions. The adjacent-run check already permits runs separated by framing tokens.
Scope
In scope: video.rs, qwen_vl.rs, vlm_runtime.rs, the worker video path, the four Qwen3-family runtimes' prompt_info, tests beside each.
Out of scope: Qwen2-VL / Qwen2.5-VL second_per_grid_ts behavior, Qwen3-Omni (separate runtime), image paths.
Acceptance Criteria
-
_video_group_timestampsport test: indices[0, 15, 30, 45, 60]at 30 fps withtemporal_patch_size = 2give[0.25, 1.25, 2.0]. - Expansion test with a stub
encode_text: one video withgrid_thw = (3, 4, 4), merge 2, yields three<t seconds>/start/4 tokens/end blocks in order; a Qwen2.5-VLprompt_infoyields the old single block unchanged. - MRoPE test: the split grids produce temporal positions that restart per group after the timestamp text, and total length matches the token count.
- Integrated:
mlxcel generate -m mlx-community/Qwen3-VL-4B-Instruct-4bit --video <clip of one moving square> -p "Describe the motion"describes a single moving object; post the before/after outputs in the PR.
Verification
cargo test --profile test-fast -p mlxcel qwen_vl
cargo test --profile test-fast -p mlxcel multimodal::video
cargo test --workspace --profile test-fast --features metal,accelerate
cargo clippy --workspace --all-targets -- -D warnings
cargo fmt --check
Activity
docs: note per-group video timestamps for the Qwen3-VL family (lablup…
hephaex commentedon Nov 9, 2026
PR #2910 renders per-group
<t seconds>timestamps and per-group MRoPE grids for the Qwen3-VL family (Qwen3-VL, Qwen3-VL-MoE, Qwen3.5 VLMs) on the server video path. Qwen2-VL and Qwen2.5-VL output is unchanged. The integrated checkpoint check from this issue is not done yet (no local Qwen3-VL checkpoint); see the PR test plan.refactor(multimodal): extract Qwen video timestamp resolution and tes…
fix: harden Qwen video timestamp validation
feat(multimodal): timestamp Qwen3-VL video groups (lablup#2910)