Skip to content

Linux video pipeline: GPU-rendered capture, tearing and audio fixes - #1621

Open
denvoros wants to merge 4 commits into
PokemonAutomation:mainfrom
denvoros:linux-video-pipeline
Open

denvoros wants to merge 4 commits into
PokemonAutomation:mainfrom
denvoros:linux-video-pipeline

Conversation

@denvoros

Copy link
Copy Markdown
Contributor

Makes video and audio capture usable on Linux. Tested on Arch with Wayland, NVIDIA, Qt 6.11 and a MiraBox capture card.

  • Fixes tearing on Wayland by forcing vsync.
  • Fixes YUYV rendering as a solid red frame on NVIDIA by defaulting v4l2src to mmap.
  • Exposes the GPU-rendered qt6-QQuickWidget backend on Linux; it uses about half the CPU of the current default. Adds QML, QVideoWidget and OpenCV backends as alternates.
  • Copies frames out of gstreamer's buffer pool before keeping them, which stops memory from growing without bound.
  • Opens audio before video, because some capture cards can't start audio once video is streaming. The audio watchdog now backs off instead of resetting every 5 seconds forever.
  • Build fixes for Qt 6.11, OpenCV 5 and GCC 13+.

NOTE: This also has one audio change for ALL Platforms!

One thing that I noticed when working on this with Claude was that some UVC capture cards (like the MiraBox i was playing with) exposes video and audio as two interfaces of one USB device and cannot start audio once video is already streaming. I got loads of usb_set_interface failed -110 errors. Qt reports the audio source as Active but delivers no data.

To fix this, I went ahead and had ConsoleSystemSession open audio before video. This changes the startup order for every platform and every capture card. Additionally, the AudioSession watchdog does 3 quick resets and then one per minute, with a one-time log message explaining why. Before that, it reset every 5 seconds which was pretty crazy.

This should be tested on Windows and Mac to make sure it doesn't break anything on those systems please.


Open questions: should Linux default to QQuickWidget, and should the OpenCV backends (about 13x the CPU) be beta-only?

Qt 6.11 deprecates UnderrunError (it is no longer emitted), which fails
-Werror builds. The default branch already covers it.
- Accept opencv5 in pkg_search_module.
- OpenCV 5 moved minAreaRect/boxPoints to opencv2/geometry.hpp; include it
  only when CV_VERSION_MAJOR >= 5 so OpenCV 4 builds are unaffected.
- WaterfillUtilities.h: include <cstdint> explicitly.
- StateReidentifier: remove unused-but-set counter.
Some UVC capture cards (e.g. MiraBox) expose video and audio as two
interfaces of one USB device and cannot start the audio stream once video
is streaming (usb_set_interface failed -110). QAudioSource reports Active
and delivers zero bytes. Opening audio first works.

- ConsoleSystemSession constructs and loads AudioSession before VideoSession.
- AudioSession watchdog: 3 quick resets, then one per minute, with a one-time
  explanation, instead of resetting every 5 seconds forever.
…tion fix

- Main.cpp (Linux): default QT_GSTREAMER_V4L2SRC_IOMODE=2 (mmap) so drivers
  that reject the capture fourcc as an EGLImage (NVIDIA + YUYV) don't render a
  flat color; force swap interval 1 to stop tearing on Wayland; warn when
  uvcvideo nodrop=1 will deliver torn frames.
- QQuickWidget backend: make the analysis connection once instead of twice
  (the extra deep copy per frame was pure waste), log its own backend name. Exposed on Linux without beta/developer mode.
- New Linux backends: qt6-QML, qt6-QVideoWidget, opencv-QOpenGLWidget,
  opencv-QWidget. VIDEO_BACKEND is saved as a slug, so adding entries is safe.
- deep_copy_frame(): copy frames out of the gstreamer buffer pool before they
  are retained by the cache/listeners (otherwise the pool grows unbounded).
- LOG_VIDEO_PATH_STATS option (default off): once-a-second log of format,
  handle type, color space/range and copy cost.
- VideoSession: log the source being switched to and when none is created.
@Mysticial

Copy link
Copy Markdown
Collaborator

I think we should wait on this one until the audio pipeline refactor is finished.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants