Skip to content

feat: Render per-group video timestamps for the Qwen3-VL family #2402

Closed

Description

@inureyes

Problem / Background

The reference Qwen3-VL processor renders a video as one <t.t seconds> text marker plus its own <|vision_start|>...<|vision_end|> block per temporal patch group, and splits each video_grid_thw row (t, h, w) into t rows of (1, h, w) for MRoPE. mlxcel renders one flat, timeless video block. Upstream found that without the markers the model reads a clip as a spatial collage (a moving square described as "several squares"); Neywa-Labs/mlx-vlm#2142 added the per-group rendering and Neywa-Labs/mlx-vlm#2413 took the timestamps from the frames actually sampled rather than an assumed 2 fps. Reference: _video_group_timestamps and _timestamped_video_placeholder in https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/qwen3_vl/processing_qwen3_vl.py, and the get_rope_index split in https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/qwen3_vl/language.py.

Current Behavior

  • insert_qwen_vl_media_tokens (src/multimodal/qwen_vl.rs:558-700) expands each video placeholder into one run of grid_t * (h/m) * (w/m) video tokens framed by a single start/end pair. It works on token ids and has no tokenizer.
  • qwen_vl_mrope_positions_from_tokens (qwen_vl.rs:949-1073) treats each video run as one (t, h, w) grid and advances current_pos by max(t, h/m, w/m).
  • The server video path (src/server/model_worker.rs:3440-3492, also used by mlxcel generate on the engine) calls video::load_video_source_with_policy (src/multimodal/video.rs:1350-1384), which computes uniform_indices(meta.total_frames, nframes) and meta.fps but returns only Vec<DynamicImage>, so the timing is lost. Qwen2VLMediaInput::Video (src/vision/processors/qwen2_vl.rs:138-141) carries frames only.
  • The same processor and expansion serve Qwen2-VL and Qwen2.5-VL, whose reference processors do not emit timestamps.

Proposed Solution

  1. Timing. Add pub struct SampledVideo { frames: Vec<DynamicImage>, frame_indices: Vec<usize>, source_fps: f64 } and load_video_source_sampled_with_policy in video.rs; make load_video_source_with_policy a thin wrapper returning .frames. The worker uses the new function and keeps the timing beside the frames.
  2. Timestamps. Port _video_group_timestamps: pad frame_indices by repeating the last index to a multiple of temporal_patch_size (the processor's field, qwen2_vl.rs:32), frame time is index / source_fps, group time is the mean of the group's first and last frame time. Error if the padded count differs from grid_t * temporal_patch_size or source_fps is not positive and finite.
  3. Family flag. Add video_timestamps: bool to QwenVlmPromptInfo (qwen_vl.rs:28-34), true for Qwen3-VL, Qwen3-VL-MoE, Qwen3.5-VLM and Qwen3.5-MoE-VLM, false for Qwen2-VL and Qwen2.5-VL (their output stays byte-identical).
  4. Expansion. compute_qwen_vl_media_embeddings (src/multimodal/vlm_runtime.rs:942) gains video_timings: &[Vec<f64>] (group seconds per video) and encode_text: Option<&dyn Fn(&str) -> anyhow::Result<Vec<i32>>>; the worker passes its tokenizer's encode without special tokens. When video_timestamps is set, each video placeholder (including its surrounding start/end pair) becomes, per group g, encode_text(format!("<{:.1} seconds>", t_g)), start, (h/m)*(w/m) video tokens, end. A timestamp family with encode_text == None is an error, not a silent fallback. The already-expanded-prompt validation path must accept this shape.
  5. MRoPE. For timestamp families, expand each video grid (t, h, w) to t grids (1, h, w) before qwen_vl_mrope_positions_from_tokens, so each group run is positioned on its own and the timestamp text between runs gets text positions. The adjacent-run check already permits runs separated by framing tokens.

Scope

In scope: video.rs, qwen_vl.rs, vlm_runtime.rs, the worker video path, the four Qwen3-family runtimes' prompt_info, tests beside each.

Out of scope: Qwen2-VL / Qwen2.5-VL second_per_grid_ts behavior, Qwen3-Omni (separate runtime), image paths.

Acceptance Criteria

  • _video_group_timestamps port test: indices [0, 15, 30, 45, 60] at 30 fps with temporal_patch_size = 2 give [0.25, 1.25, 2.0].
  • Expansion test with a stub encode_text: one video with grid_thw = (3, 4, 4), merge 2, yields three <t seconds>/start/4 tokens/end blocks in order; a Qwen2.5-VL prompt_info yields the old single block unchanged.
  • MRoPE test: the split grids produce temporal positions that restart per group after the timestamp text, and total length matches the token count.
  • Integrated: mlxcel generate -m mlx-community/Qwen3-VL-4B-Instruct-4bit --video <clip of one moving square> -p "Describe the motion" describes a single moving object; post the before/after outputs in the PR.

Verification

cargo test --profile test-fast -p mlxcel qwen_vl
cargo test --profile test-fast -p mlxcel multimodal::video
cargo test --workspace --profile test-fast --features metal,accelerate
cargo clippy --workspace --all-targets -- -D warnings
cargo fmt --check

Activity

  1. added
    type:enhancementNew features, capabilities, or significant additions
    area:modelsModel architectures, weights, loading, metadata
    on Oct 16, 2026
  2. hephaex commented on Nov 9, 2026

    @hephaex
    Contributor

    PR #2910 renders per-group <t seconds> timestamps and per-group MRoPE grids for the Qwen3-VL family (Qwen3-VL, Qwen3-VL-MoE, Qwen3.5 VLMs) on the server video path. Qwen2-VL and Qwen2.5-VL output is unchanged. The integrated checkpoint check from this issue is not done yet (no local Qwen3-VL checkpoint); see the PR test plan.

  3. added a commit that references this issue on Nov 13, 2026
    28d884a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:modelsModel architectures, weights, loading, metadatamodeltype:vlmVision-language modelpriority:mediumMedium prioritystatus:doneCompletedtype:enhancementNew features, capabilities, or significant additions

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions