Skip to content

feat(📸): draw view snapshots into the texture on the GPU on Android 13+ - #527

Open
wcandillon wants to merge 13 commits into
mainfrom
feat/android-view-snapshot-zero-copy
Open

wcandillon wants to merge 13 commits into
mainfrom
feat/android-view-snapshot-zero-copy

Conversation

@wcandillon

Copy link
Copy Markdown
Owner

Stacked on #526 (which is stacked on #524). The base is the #526 branch; GitHub retargets it once that merges.

Why

drawElementImageToTexture on Android renders the view through HardwareRenderer into a GPU-sampleable buffer, then reads it back (Bitmap.wrapHardwareBuffer + copy(ARGB_8888)), copies the pixels again on the native side and uploads them with writeTexture. The code already noted that the buffer could be imported directly. The missing piece was synchronization: HardwareRenderer only submits the frame, and the fence that says when the drawing is done is Image.getFence() (API 33), the same fence Dawn's Android layer feeds into beginAccess. #524 added the plumbing for that fence.

What

  • Zero-copy path (Android 13+, texture with RENDER_ATTACHMENT). The renderer's AHardwareBuffer goes to native with its fence (onViewSnapshotBuffer, which takes a reference to the buffer and owns the descriptor). The queue imports it as an external texture, so BeginAccess waits on the fence on the GPU, and draws it into the destination region with the blit that backs copyExternalImageToTexture (viewport and scissor confine the render pass to destination.origin + size). The blit gains an alpha mode: opaque (video, as before), keep premultiplied, or straighten (the premultipliedAlpha: false default), applied in the encoded domain like the CPU path.
  • Everything else is unchanged. Without RENDER_ATTACHMENT, before API 33, on iOS, or when the hardware render fails, the pixels are read back as before. COPY_DST stays required so the fallback always works; RENDER_ATTACHMENT is additive. The docs say so.
  • Plumbing. ViewSnapshotRequest.allowFrame; PlatformContext::snapshotView(request, onPixels, onFrame, onError); ViewSnapshot.renderHardware factors the HardwareRenderer setup so both consumers (readback, hand-over) share it. A blit failure on the runtime thread rejects the promise by resolving it with a rejected JS promise (the resolve factory runs with no try/catch and cannot reject itself).
  • Specs. Two new ViewSnapshot tests draw into renderable textures: the probe view at natural size, and the lime quadrant into a 16x16 region of a 32x32 bgra8unorm texture whose rest must stay zero. On the API 35 emulator they take the GPU path; elsewhere the fallback, with the same expectations.

Verification

  • Android: :react-native-webgpu:assembleDebug (arm64-v8a), CMake and Java compile.
  • iOS: the react-native-webgpu pod target builds for the simulator (signature change only).
  • yarn tsc, yarn lint, cpplint on the touched files.
  • Not run on a device here; the two new specs are the on-device check (CI runs them on the API 35 emulator).

…ideo frames

ImageReader hands a buffer out as soon as the producer queues it and only
waits on the producer's fence for CPU access (Image.getPlanes), not for
Image.getHardwareBuffer. The GPU could therefore sample a frame the video
decoder (or the camera) was still writing.

Since API 33 the fence is exposed as Image.getFence(). SyncFences reads
its file descriptor (SyncFence has no accessor; a Parcel round-trip yields
a duplicate), the frame handle carries it, and GPUExternalTexture::Create
imports it as a shared-fence-sync-fd fence that BeginAccess waits on, on
the GPU. A device without that feature waits on the CPU instead; before
API 33 nothing changes.

The fallback compileSdkVersion of the library goes to 34 so the SyncFence
API compiles when the app does not set one (every current React Native
app does).
…frame's color space

textureSampleBaseClampToEdge now returns RGB on Android as it does on iOS
and the web. The Android path used to hand the shader raw Y, Cb, Cr plus a
non-spec GPUExternalTexture.yuvToRgbMatrix to apply itself: Dawn's Tint
transform skipped the conversion matrix for single-plane external
textures. The Dawn this package ships (m154) carries Dawn's YUV
AHardwareBuffer specialization, which applies the descriptor's matrix,
transfer functions and gamut conversion while sampling an
OpaqueYCbCrAndroid plane, so the matrix goes in the descriptor instead and
the shader-side workaround and its getter go away.

The parameters come from dawn::ComputeExternalTextureParams, the helper
Chromium feeds video frames through. Its header is vendored under
cpp/dawn/common (the binaries export the function; the package does not
ship the header) and pinned to the Dawn commit in scripts/check-dawn.ts.
Its input is the frame's color space, which now travels with the frame:
the CVPixelBuffer attachments on Apple platforms (matrix, range,
primaries, transfer), Image.getDataSpace() (API 33) or the container's
color keys on Android. What a producer leaves unknown falls back to what
the platform reports for the buffer (the Vulkan driver's suggested YCbCr
model and range), then to BT.709 limited range on Apple platforms and
BT.601 limited range, the camera norm, on Android. The descriptor's spec
colorSpace member ("srgb" | "display-p3") selects the output.

createTestVideoFrame(w, h, "nv12") builds a BT.709 limited-range YUV
frame on both platforms (red left, green right); a spec imports it and
checks the colors that come back.
…d friends)

Image.getDataSpace() can still return the pre-bitfield values from
system/graphics.h: a Pixel 8 on Android 17 tags every H.264 decoder frame
with HAL_DATASPACE_BT709 (0x104), which the standard/transfer/range masks
read as unknown. Map the six legacy values to the field form the NDK header
documents for them before decoding.
drawElementImageToTexture rendered the view through HardwareRenderer into
a GPU buffer, then read it back into a software bitmap, copied the pixels
again on the native side and uploaded them with writeTexture. The buffer
can be imported instead: a HardwareRenderer frame comes with the fence
that signals when the drawing is done (Image.getFence, API 33), which
BeginAccess waits on, on the GPU.

When the destination texture has RENDER_ATTACHMENT, Android 13 and above
hand the renderer's AHardwareBuffer to native with that fence, and the
view is drawn into the destination region by the external-texture blit of
copyExternalImageToTexture (which learns to keep or straighten the
frame's premultiplied alpha). Without RENDER_ATTACHMENT, before API 33,
on iOS, or when the hardware render fails, the pixels are read back as
before, so COPY_DST stays required and the results are identical.

The snapshot request carries whether the caller can take a GPU frame, the
platform answers with pixels or a frame, and a blit failure on the
runtime thread rejects the promise through a rejected JS promise, since a
resolve factory cannot reject by itself. Two specs draw into renderable
textures and check the pixels, one of them a region of a bgra8unorm
texture whose rest has to stay untouched.
Dawn's TexelCopyTextureInfo defaults aspect to Undefined (read as "all"),
and the converter leaves it there when the descriptor has no aspect, so the
RENDER_ATTACHMENT hand-over was only selected with an explicit aspect: "all".
Found on a Pixel 8: every drawElementImageToTexture call reached Java with
allowHardwareBuffer=false.
@wcandillon

Copy link
Copy Markdown
Owner Author

Device round on a Pixel 8 (Android 17):

  • Found and fixed (3a6a074): the GPU hand-over was never selected unless the caller passed aspect: "all" explicitly. Dawn's TexelCopyTextureInfo defaults aspect to Undefined and the converter leaves it there when the descriptor omits it, so dst.aspect == All was false; every call reached Java with allowHardwareBuffer=false. The spec passed anyway because the readback fallback produces the same pixels. The check now uses the descriptor's optional with "all" as the default.
  • After the fix, both renderable-texture tests take the hand-over: the HardwareRenderer buffer arrives with a live fence fd, Dawn imports it as a shared fence, and the pixel checks still pass. The View Snapshot example with RENDER_ATTACHMENT added renders the card correctly (premultiplied alpha, ripple).
  • Full e2e suite: 43 suites, 214 tests pass on the device.

wcandillon and others added 6 commits October 15, 2026 08:30
…-texture-color-space

# Conflicts:
#	packages/webgpu/android/cpp/AndroidVideoPlayer.h
#	packages/webgpu/cpp/rnwgpu/PlatformContext.h

Co-authored-by: wcandillon <306134+wcandillon@users.noreply.github.com>
@wcandillon
wcandillon changed the base branch from feat/android-external-texture-color-space to main October 15, 2026 07:11

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants