feat(layers): composition layer support behind enableLayers() - #516
Open
salmanmkc wants to merge 14 commits into
Open
feat(layers): composition layer support behind enableLayers()#516salmanmkc wants to merge 14 commits into
salmanmkc wants to merge 14 commits into
Conversation
Layers are optional in two independent places, and a session existing says nothing about either: the session may not have been granted the layers feature, and the binding may not offer the layer type wanted. The two backings are worth telling apart rather than reducing to a boolean. XRMediaBinding lets the compositor drive a video element with the app never drawing a frame for it, which is the cheap path, but Quest is the only browser that ships it. Chrome has quad layers only through XRWebGLBinding, where the app uploads each frame itself. Anything that wants to run in both places has to know which one it is on. preferWebGL exists because Quest has both and would otherwise always take the media path, leaving the WebGL path with no hardware to run on.
three.js sets layers to a single projection layer once when the session starts and never touches that array again. Its later updateRenderState calls only carry depthNear and depthFar, which merge, so adding to the array is safe, but the array has to be rebuilt in full every time and leaving the projection layer out of it blanks the scene. That is a sharp edge to hand to callers, so the manager owns it: nothing can be added before setBaseLayer has been told what three.js is rendering into, and every submission puts that layer first. Ordering follows the spec, earlier entries composite behind later ones, so app layers sit in front of the scene. The session, binding and context are held together because the WebGL path needs all three to upload a frame, and dropped together when the session ends so nothing outlives it.
Quad layers need the layers feature named when the session is requested. It cannot be turned on afterwards, so a session that did not ask for it can never have one, and createQuadLayer throws NotSupportedError rather than degrading. It goes in as an optional feature, not a required one, so asking for layers on a platform without them still gives a working session that falls back to drawing into the scene. Off by default, since it costs a feature request that most apps have no use for.
Video is the case where going through the eye buffer costs the most. A 1.2m quad two metres away covers roughly a third of the field of view, so on a Quest 3 panel it lands on about 620 pixels across. Hand it a 2048 wide source and around 70 percent of the detail is gone before the compositor ever sees the frame, and what survives gets resampled a second time when it warps to the head pose. As a composition layer it is sampled once, at its own resolution. This is the XRMediaBinding path, where the compositor drives the video element and the app never draws a frame for it. VideoLayer reports whether it got a layer rather than throwing when it cannot, so an app can ask for one everywhere and draw the video into the scene on platforms that have none. Each way it can fail falls back separately: no session, no base layer to compose against, no media binding, and a platform that advertises the binding but still refuses a particular element, which happens with one whose metadata has not loaded yet. Quad extents need halving here and it looks like a bug without the reason. The spec means width and height as full metres and Chromium passes them to OpenXR unchanged, but the Quest compositor treats them as half extents and returns a quad at twice the size. It is a property of the compositor rather than of this binding, so it is decided by the platform. See immersive-web/layers#324, still open. Height comes from the video's aspect ratio unless given, defaulting to 16:9 while metadata is still loading rather than dividing by a zero videoWidth.
Chrome does not implement XRMediaBinding and is not going to, so every Android XR device falls back to drawing the video into the scene, which is the case this whole thing exists to avoid. This adds the path Chrome does have: create the quad through XRWebGLBinding and upload each frame into its texture with texSubImage2D. The layer texture is allocated once, at the size given when it is created, and texSubImage2D refuses a source that does not fit it. Guessing a size and later uploading a frame of a different one raises a GL error on every frame, which on a Quest took the whole session down, so it waits for the video to report real dimensions instead. Binding a texture directly also has to be undone. three.js caches GL state and skips calls it believes are redundant, so leaving the layer texture bound left its cache describing something untrue and it went on to render with the wrong texture. The active unit, the 2D binding and the flip flag are all put back. Uploads happen only for frames the video has actually advanced past. The source runs at 30fps and the headset renders at 90, so two in three were pushing an identical 2048x1152 image, which is about 850MB/s of texture traffic for nothing. needsRedraw still forces one, since that means the compositor lost the layer contents. Layers are created with isStatic false, because getSubImage raises InvalidStateError on a static layer once needsRedraw has gone false and a video needs a new frame every time. Layout is named explicitly since 'default' is a TypeError for quad layers. getSubImage also throws until updateRenderState has taken effect, a frame or two after the layer is added, so a failed upload drops that frame rather than tearing the layer down.
The first attempt at this demo used a sample video off a CDN. Both copies looked identical, which was not the comparison working, it was the source being lower resolution than the eye buffer already: there was nothing left for the layer path to preserve. So the card is generated instead, at 2048x1152 onto a canvas and fed to both sides through captureStream. Text runs from 96px down to 12px and there are one, two and three pixel line pairs, which is where resampling shows first and where the eye buffer path visibly loses. Being generated also means no asset to host, and the resolution can be changed to find where the difference stops mattering.
Puts the same generated card up twice, once as a composition layer and once as an ordinary textured plane in the scene, so the only variable is the path the frame takes to the display. The two sides swap on a timer, which is load bearing rather than a gimmick. Lenses are sharpest in the middle and you read one panel at a time, so a fixed left and right layout cannot separate "the layer is sharper" from "that side happened to be in the sweet spot". Swapping does separate them: a difference that follows the path is real, one that stays on a side is the optics. Select swaps early and deliberately does not stop the timer. Squeeze is bound to select end, so a grip press while holding the controller ended the comparison for good, with nothing on screen to say why. The readout says which binding was asked for, which one is actually running and how many frames have been uploaded, because a line that only says "media" cannot distinguish a request that was ignored from one that was never made. It renders to the page and into the session both, since a DOM overlay disappears the moment you enter XR. ?webglLayer=1 forces the WebGL path on a platform that has both, which is what lets a Quest stand in for Android XR.
Wrapped returns and argument lists the formatter wanted broken differently. No behaviour change.
dli7319
self-requested a review
August 13, 2026 00:47
Collaborator
|
Both sides are looking the same to me on Galaxy XR. Hmm. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
I added webgl support so this should work on android xr too, I tested that on meta quest, though not sure if dimensions need adjusting on android xr and if it works fully there. The demo shows switching between layer on and off. Webgl looks the exact same on my meta quest.
layer.demo.mp4
Anything drawn into the eye buffer is resampled at least twice before you see it: once into the buffer, which is already coarser than the panel across the area a given object covers, and again when the compositor warps it to head pose. A quad composition layer skips both and is sampled once, at its own resolution, straight by the compositor. That is a property of the render path rather than of any one headset.
The cost is easiest to see with numbers. A 1.2m quad two metres away covers about a third of the field of view, so on a Quest 3 it lands on roughly 620 eye buffer pixels across. Feed that a 2048 wide source and around 70% of the detail is gone before the compositor has even seen the frame. Fine text and single pixel lines are where it shows.
Adds
LayerManager,LayerCapabilityandVideoLayerundersrc/layers/, plus a demo indemos/layers/. All behindoptions.enableLayers(), off by default. Split into seven commits since it introduces a subsystem, and each one stands on its own.Both backings are implemented, because neither one covers the devices we care about on its own.
XRMediaBindingis the cheap path, where the compositor drives the video element and the app never draws a frame, but Quest is the only browser that ships it. Chrome has quad layers only throughXRWebGLBinding, where the app uploads each frame itself, so that is the path Android XR needs.LayerManagerexists because three.js setslayers: [projectionLayer]once at session start and never touches that array again. Anything extra has to be composed back in together with the projection layer, and dropping it blanks the scene, sosetBaseLayeris required before anything can be added.VideoLayerreports whether it got a layer instead of throwing, so an app can ask for one everywhere and just draw into the scene where there isn't one. It falls back separately for no session, no base layer, no binding, and a platform that advertises a binding but refuses a particular element.The quad sizing is worth calling out, because the two paths deliberately differ and it looks like a bug otherwise. The spec means
widthandheightas full metres, and Chromium passes them to OpenXR unchanged, but the Quest compositor treats them as half extents and returns a quad at twice the size. It is a property of the compositor rather than of one binding, so the correction is decided by the platform, not the path. This is immersive-web/layers#324, still open. Worth knowing that three.js and A-Frame both hardcode a/2for Quest, which means copying either of them renders the quad at half size on Android XR.A few other things the spec is explicit about and the official samples get wrong or omit:
layouthas to be named because'default'is aTypeErrorfor quad layers,isStatichas to be false orgetSubImageraisesInvalidStateErroronceneedsRedrawclears, andgetSubImagethrows untilupdateRenderStatehas taken effect a frame or two after the layer is added, so a failed upload drops that frame rather than tearing the layer down.Two faults only showed up on device. The layer texture is allocated once at the size given at creation, and the test card is a canvas stream that reports no dimensions for its first frames, so guessing a size and then uploading a differently sized frame raised a GL error every frame and took the whole session down. And uploading through raw
gl.bindTextureleft three.js's state cache describing something untrue, so it carried on rendering with the compositor's texture bound. Uploads are also skipped for frames the video has not advanced past, since the source runs at 30fps against a 90fps render loop.The demo puts the same generated card up twice, once as a layer and once as a plane in the scene, so the only variable is the path the frame takes. It is generated rather than a video file because the sample clips I tried are lower resolution than the eye buffer, so both sides looked identical and it showed nothing. The two sides swap on a timer, which matters: lenses are sharpest in the middle and you read one panel at a time, so a fixed left/right layout cannot separate "the layer is sharper" from "that side was in the sweet spot". Swapping does separate them, and on device the difference follows the layer across the swap.
?webglLayer=1forces the WebGL path on a platform that has both, which is what lets a Quest stand in for Android XR.Type of Change
Media / Screen Recordings & Screenshots (If Applicable)
Checklist