Inference quality question

[kingdeng05]

Hi,

Thanks for open sourcing this work and the pipeline for the community to try out and build upon.

I ran the public mint_v1 checkpoint (checkpoints/model.safetensors, mint_step2.yaml) on the bundled data/samples/lerobot_v3 HOT3D sample in the Web Viewer.

Load Model then Start Inference, hand merge Hard Cut (no UKF). GT overlay sits on the real hands; Pred is clearly offset (mesh/skeleton not on the hand). Same frame index, offline processing.

Is this expected accuracy for the released checkpoint on this sample, or are we missing a setting/weight? Photo attached.

[ZZJ-Asher]

Thanks for the detailed report and screenshot.
Regarding the 2D projection offset you observed, I think this can be discussed from two aspects:
FOV estimation error: The model output includes an FOV estimate, and the FOV itself is predicted with some error. Since FOV directly affects the 2D projection result, even a small FOV inaccuracy can lead to a noticeable offset in the projected 2D coordinates.
Limitations of 2D projection as an evaluation metric: 2D projection overlap only indicates that the predicted XY direction is roughly correct. Methods like Wilor that first detect a bounding box (e.g., with YOLO) and then predict the hand within that box can ensure the hand stays aligned in 2D video, but they suffer from a different failure mode — if the detector misidentifies the box, the hand prediction can severely drift. In our case, the predicted 3D hand may actually be quite close to the ground truth in 3D space, but when projected to 2D, even a modest 3D error can translate into a visible 2D offset, especially under perspective projection.