[hephaex]

Summary

The reference Qwen3-VL processor renders a video as one <t.t seconds> marker plus its own <|vision_start|>...<|vision_end|> block per temporal patch group, and positions each group as a (1, h, w) MRoPE grid. mlxcel rendered one flat, timeless video block, which the model reads as a spatial collage. This renders per-group timestamps for the Qwen3-VL family on the server video path, using the times of the frames the sampler actually kept (Neywa-Labs/mlx-vlm#2142, #2413).

Related issues

Closes #2402

Type of change

• feat — new user-visible feature

What changed

• multimodal/video.rs: SampledVideo { frames, frame_indices, source_fps } and load_video_source_sampled_with_policy[_and_budget]. The existing loaders are thin wrappers returning .frames. If the decoder returns fewer frames than requested, frame_indices is truncated to match.
• multimodal/qwen_vl.rs:
  • QwenVlmPromptInfo::video_timestamps gates the behavior: true for Qwen3-VL, Qwen3-VL-MoE and Qwen3.5 VLMs; false for Qwen2-VL, Qwen2.5-VL, GLM, Cohere Compass, Qwen4-Exp and MiMo. Their token output is unchanged.
  • qwen_video_group_timestamps ports _video_group_timestamps: indices are padded to a multiple of temporal_patch_size by repeating the last one, a frame's time is index / source_fps, a group's time is the mean of its first and last frame time. It errors on a non-positive/non-finite fps or when the padded count does not match grid_t * temporal_patch_size.
  • insert_qwen_vl_media_tokens_timestamped emits <ts> <start> <video>* <end> per group. It handles the same three prompt states as the flat layout: no placeholders, one placeholder per item (the template's own start/end pair is folded in, not doubled), and already-expanded prompts (flat or per-group runs validated).
  • qwen_vl_mrope_positions_from_tokens reads a video run that is exactly one group long as the first of t per-group (1, h, w) runs, so the temporal axis restarts after each timestamp. Flat runs behave as before.
• multimodal/vlm_runtime.rs: compute_qwen_vl_media_embeddings takes Option<&QwenVideoTimestamps>. A timestamp family with videos and no timing is an error, not a silent fallback to the flat layout.
• server/model_worker.rs: prepare_qwen_vl_video_embeddings takes the tokenizer, encodes <t seconds> without special tokens, and computes group times from the sampled indices.
• docs/supported-models.md: one paragraph noting the new rendering.

Test plan

• cargo fmt --check
• cargo clippy -p mlxcel --all-targets -- -D warnings (Metal off, macOS arm64)
• cargo test --profile test-fast -p mlxcel --features accelerate --lib qwen_vl — 29 passed. New tests cover: MRoPE temporal positions restarting per group after the timestamp text and total length matching the token count; missing/malformed group runs rejected; already-expanded prompts accepted; one framed run per group; images kept flat and mixed media ordered; insertion after BOS with no placeholders; mean-of-first/last timestamps; one-decimal text; inconsistent inputs rejected.
• cargo test --profile test-fast -p mlxcel --features accelerate --lib -- multimodal::qwen_vl multimodal::vlm_runtime multimodal::video — 124 passed, 13 ignored. The timestamp-resolution step of compute_qwen_vl_media_embeddings is the pure fn qwen_video_group_prefixes, unit-tested without a QwenVlRuntime (missing timing, video/group count mismatch, tokenizer error, non-timestamp family, success).
• cargo test ... --lib -- --include-ignored multimodal::video (ffmpeg present) — 53 passed, including the new sampled_loader_reports_source_indices_and_fps. The 1 failure, bench_single_pass_768_frames, is a wall-clock benchmark that its own doc says is not a gate.
• Not run: the integrated check from the issue (Qwen3-VL-4B-Instruct-4bit on a clip of one moving square, before/after). No Qwen3-VL checkpoint is available locally. Reviewers with one, please run it. This is synthetic-only validation of the token layout and MRoPE positions, not model quality.
• cargo test --workspace --features metal,accelerate was not run. I ran --lib with Metal off: 11393 passed / 17 failed. The failures are CPU-only-environment or flaky cases; I re-ran three of them (qwen4_exp ... qk_normalization_matches_upstream, catalog_cache_uses_epoch..., sam3d_objects ... compiled_objects_gelu_tanh...) on the unmodified base and they fail identically there. batched_lookahead_matches_force_sync_past_the_window passes and fails intermittently on repeated runs without my change.
• cargo clippy --workspace fails on mlxcel-xla/src/emitter/builder.rs:374 (manual_is_multiple_of), a pre-existing lint in code this PR does not touch.

Notes for reviewers

• The mixed flat/per-group detection in MRoPE relies on a flat run for t > 1 never being exactly one group long, so it only reinterprets what was previously a cardinality error.
• mlxcel generate --video expands clips to frames through a separate path (expand_cli_videos_to_frames); this PR changes the server video path only.
• Timestamp text is encoded with encode(text, false); the marker tokenization relies on the model's own tokenizer matching the reference <%.1f seconds> text.

Checklist

• PR title uses a Conventional Commits prefix
• One logical change per PR
• Updated docs/ (user-facing behavior)
• No secrets, credentials, or .env files committed

Hardening verification (2026-11-11)

The original submission's unmeasured integrated test and server-only scope note are superseded: native mlxcel generate --video uses the shared server preparation path. Real Qwen3-VL-4B moving-square inference now describes a single square moving left to right; before/after outputs and reproduction are recorded in TECHNICAL_REPORTS/2910-qwen-video-timestamp-hardening-20261111.en.md and its Korean companion.

Commit 613247d removes timestamp arithmetic overflow and padding allocation hazards, rejects unordered indices and non-finite/negative times, and adds boundary regressions and upstream attribution. Local Metal tests passed (126 focused tests and 53 ffmpeg tests), as did Clippy and formatting; executed PR CI checks passed. Self-hosted fork-PR jobs were skipped. Llama vision fallback text was identical before/after but hallucinated content in both arms. Full workspace tests and additional Qwen-family checkpoints remain unmeasured.

[inureyes]

Hardening review

Reviewed PR #2910 against issue #2402, its discussion, and the upstream timestamp implementation. Commit 613247d integrates main and fixes unchecked dimension multiplication, potentially oversized padding allocation, unordered frame indices, and non-finite/negative timestamps. Added boundary regressions and the upstream MIT derivation notice. No new network resolution, subprocess path, shared mutable state, or unbounded cache is introduced; timestamp calculation stays linear without a padded copy.

The original server-only scope note is inaccurate: native Qwen video already uses the shared preparation path from mlxcel generate --video. The documentation now states this explicitly.

Validation

• cargo test --profile test-fast --features metal,accelerate -p mlxcel --lib -- multimodal::qwen_vl multimodal::vlm_runtime multimodal::video --test-threads=1: 126 passed, 13 ignored.
• The ffmpeg video suite with --include-ignored --skip bench_single_pass_768_frames --test-threads=1: 53 passed. Only the explicitly non-gating timing benchmark was excluded.
• Metal-enabled Clippy (--profile test-fast -p mlxcel --lib --tests -- -D warnings), formatting, and whitespace checks passed.
• Real Qwen3-VL-4B-Instruct-4bit on M1 Ultra, identical moving-square video and greedy prompt: before described a motion-blur effect; after answered “A white square moves from the left side of the screen to the center, then to the right side of the screen.” Full reproduction and before/after output are in the bilingual technical report.
• Llama 3.2 11B vision fallback generated identical text before/after without crashing. Both outputs hallucinated content; this is regression evidence, not a model-quality pass.

Full workspace tests and real Qwen3-VL-MoE/Qwen3.5 VLM checkpoints were not run. No throughput improvement is claimed. Existing native C++ unused-capture warnings remain.