Repository navigation
feat(📸): draw view snapshots into the texture on the GPU on Android 13+ - #527
Open
wcandillon wants to merge 13 commits into
Open
wcandillon wants to merge 13 commits into
wcandillon wants to merge 13 commits into
Conversation
…ideo frames ImageReader hands a buffer out as soon as the producer queues it and only waits on the producer's fence for CPU access (Image.getPlanes), not for Image.getHardwareBuffer. The GPU could therefore sample a frame the video decoder (or the camera) was still writing. Since API 33 the fence is exposed as Image.getFence(). SyncFences reads its file descriptor (SyncFence has no accessor; a Parcel round-trip yields a duplicate), the frame handle carries it, and GPUExternalTexture::Create imports it as a shared-fence-sync-fd fence that BeginAccess waits on, on the GPU. A device without that feature waits on the CPU instead; before API 33 nothing changes. The fallback compileSdkVersion of the library goes to 34 so the SyncFence API compiles when the app does not set one (every current React Native app does).
…frame's color space
textureSampleBaseClampToEdge now returns RGB on Android as it does on iOS
and the web. The Android path used to hand the shader raw Y, Cb, Cr plus a
non-spec GPUExternalTexture.yuvToRgbMatrix to apply itself: Dawn's Tint
transform skipped the conversion matrix for single-plane external
textures. The Dawn this package ships (m154) carries Dawn's YUV
AHardwareBuffer specialization, which applies the descriptor's matrix,
transfer functions and gamut conversion while sampling an
OpaqueYCbCrAndroid plane, so the matrix goes in the descriptor instead and
the shader-side workaround and its getter go away.
The parameters come from dawn::ComputeExternalTextureParams, the helper
Chromium feeds video frames through. Its header is vendored under
cpp/dawn/common (the binaries export the function; the package does not
ship the header) and pinned to the Dawn commit in scripts/check-dawn.ts.
Its input is the frame's color space, which now travels with the frame:
the CVPixelBuffer attachments on Apple platforms (matrix, range,
primaries, transfer), Image.getDataSpace() (API 33) or the container's
color keys on Android. What a producer leaves unknown falls back to what
the platform reports for the buffer (the Vulkan driver's suggested YCbCr
model and range), then to BT.709 limited range on Apple platforms and
BT.601 limited range, the camera norm, on Android. The descriptor's spec
colorSpace member ("srgb" | "display-p3") selects the output.
createTestVideoFrame(w, h, "nv12") builds a BT.709 limited-range YUV
frame on both platforms (red left, green right); a spec imports it and
checks the colors that come back.
…d friends) Image.getDataSpace() can still return the pre-bitfield values from system/graphics.h: a Pixel 8 on Android 17 tags every H.264 decoder frame with HAL_DATASPACE_BT709 (0x104), which the standard/transfer/range masks read as unknown. Map the six legacy values to the field form the NDK header documents for them before decoding.
drawElementImageToTexture rendered the view through HardwareRenderer into a GPU buffer, then read it back into a software bitmap, copied the pixels again on the native side and uploaded them with writeTexture. The buffer can be imported instead: a HardwareRenderer frame comes with the fence that signals when the drawing is done (Image.getFence, API 33), which BeginAccess waits on, on the GPU. When the destination texture has RENDER_ATTACHMENT, Android 13 and above hand the renderer's AHardwareBuffer to native with that fence, and the view is drawn into the destination region by the external-texture blit of copyExternalImageToTexture (which learns to keep or straighten the frame's premultiplied alpha). Without RENDER_ATTACHMENT, before API 33, on iOS, or when the hardware render fails, the pixels are read back as before, so COPY_DST stays required and the results are identical. The snapshot request carries whether the caller can take a GPU frame, the platform answers with pixels or a frame, and a blit failure on the runtime thread rejects the promise through a rejected JS promise, since a resolve factory cannot reject by itself. Two specs draw into renderable textures and check the pixels, one of them a region of a bgra8unorm texture whose rest has to stay untouched.
Dawn's TexelCopyTextureInfo defaults aspect to Undefined (read as "all"), and the converter leaves it there when the descriptor has no aspect, so the RENDER_ATTACHMENT hand-over was only selected with an explicit aspect: "all". Found on a Pixel 8: every drawElementImageToTexture call reached Java with allowHardwareBuffer=false.
wcandillon
force-pushed
the
feat/android-view-snapshot-zero-copy
branch
from
October 15, 2026 05:57
ccd6982 to
3a6a074
Compare
Owner
Author
|
Device round on a Pixel 8 (Android 17):
|
…rnal-texture-color-space
…droid-view-snapshot-zero-copy
…-texture-color-space # Conflicts: # packages/webgpu/android/cpp/AndroidVideoPlayer.h # packages/webgpu/cpp/rnwgpu/PlatformContext.h Co-authored-by: wcandillon <306134+wcandillon@users.noreply.github.com>
…droid-view-snapshot-zero-copy
wcandillon
changed the base branch from
feat/android-external-texture-color-space
to
main
October 15, 2026 07:11
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #526 (which is stacked on #524). The base is the #526 branch; GitHub retargets it once that merges.
Why
drawElementImageToTextureon Android renders the view throughHardwareRendererinto a GPU-sampleable buffer, then reads it back (Bitmap.wrapHardwareBuffer+copy(ARGB_8888)), copies the pixels again on the native side and uploads them withwriteTexture. The code already noted that the buffer could be imported directly. The missing piece was synchronization:HardwareRendereronly submits the frame, and the fence that says when the drawing is done isImage.getFence()(API 33), the same fence Dawn's Android layer feeds intobeginAccess. #524 added the plumbing for that fence.What
RENDER_ATTACHMENT). The renderer'sAHardwareBuffergoes to native with its fence (onViewSnapshotBuffer, which takes a reference to the buffer and owns the descriptor). The queue imports it as an external texture, soBeginAccesswaits on the fence on the GPU, and draws it into the destination region with the blit that backscopyExternalImageToTexture(viewport and scissor confine the render pass todestination.origin+ size). The blit gains an alpha mode: opaque (video, as before), keep premultiplied, or straighten (thepremultipliedAlpha: falsedefault), applied in the encoded domain like the CPU path.RENDER_ATTACHMENT, before API 33, on iOS, or when the hardware render fails, the pixels are read back as before.COPY_DSTstays required so the fallback always works;RENDER_ATTACHMENTis additive. The docs say so.ViewSnapshotRequest.allowFrame;PlatformContext::snapshotView(request, onPixels, onFrame, onError);ViewSnapshot.renderHardwarefactors theHardwareRenderersetup so both consumers (readback, hand-over) share it. A blit failure on the runtime thread rejects the promise by resolving it with a rejected JS promise (the resolve factory runs with no try/catch and cannot reject itself).ViewSnapshottests draw into renderable textures: the probe view at natural size, and the lime quadrant into a 16x16 region of a 32x32bgra8unormtexture whose rest must stay zero. On the API 35 emulator they take the GPU path; elsewhere the fallback, with the same expectations.Verification
:react-native-webgpu:assembleDebug(arm64-v8a), CMake and Java compile.react-native-webgpupod target builds for the simulator (signature change only).yarn tsc,yarn lint, cpplint on the touched files.