From 85ad077ec743095d235dd5e5343c588992ae0580 Mon Sep 17 00:00:00 2001 From: Rob Oates Date: Mon, 5 Oct 2026 01:18:07 +0900 Subject: [PATCH 01/29] Port PhoneVR client to ALVR v21 API (shared EGL context, decoder config handling) --- MIGRATION_NOTES.md | 69 ++ .../mobile/android/PhoneVR/app/CMakeLists.txt | 17 - code/mobile/android/PhoneVR/app/build.gradle | 22 +- .../PhoneVR/app/src/main/AndroidManifest.xml | 1 + .../PhoneVR/app/src/main/cpp/alvr_main.cpp | 756 +++++++++++------- .../app/src/main/res/values/strings.xml | 1 + .../android/PhoneVR/prepare-alvr-deps.sh | 12 - 7 files changed, 567 insertions(+), 311 deletions(-) create mode 100644 MIGRATION_NOTES.md mode change 100644 => 100755 code/mobile/android/PhoneVR/prepare-alvr-deps.sh diff --git a/MIGRATION_NOTES.md b/MIGRATION_NOTES.md new file mode 100644 index 00000000..1aad3f4d --- /dev/null +++ b/MIGRATION_NOTES.md @@ -0,0 +1,69 @@ +# PhoneVR: ALVR v20 -> v21 migration notes + +Generated while porting the Android client's `main` flavor (the one built on `alvr_client_core`, +not the legacy `gvr` custom-protocol flavor) from ALVR v20.8.0 to the v21 nightly +(commit `dc84d6835d4125226f9698f1f052629852d86e31`). + +## 1. Pruned (removed) + +- `code/mobile/android/PhoneVR/app/src/gvr/` and `app/src/androidTestGvr/` - the legacy, + fully custom UDP-socket protocol client (`PVRSockets`/`PVRRenderer`, Google VR SDK). Superseded + entirely by the `main` flavor, which already links `alvr_client_core` directly. +- `code/mobile/mobile-common/` - only consumed by the removed `gvr` flavor. +- `gvrImplementation 'com.google.vr:sdk-base:...'` and the GVR SDK unzip step in + `extractNdk` (`app/build.gradle`). +- The `flavorDimensions "gvr"` / `productFlavors { gvr / noGvr }` split - now a single + build variant. +- `IF(${HAVEGVR}) add_subdirectory(src/gvr) ENDIF()` in `CMakeLists.txt`, plus the now-unused + `asio`/`eigen`/`ifaddrs` include dirs (only needed by the removed custom protocol). +- The GVR SDK download block in `prepare-alvr-deps.sh`, and the `nogvr` CLI flag it existed for. +- Manual settings-JSON parsing for foveated encoding in `alvr_main.cpp` - v21 delivers + `foveated_encoding` directly on the `StreamingStarted` event, so the `nlohmann/json` dependency + is gone from this file entirely. + +**Not yet touched, but likely also obsolete now:** `driver/PVRServer` (a SteamVR driver for the +removed custom protocol's PC side) and `code/windows`. Left alone since they're outside the phone +client and weren't explicitly in scope. `.github/workflows/build.yml` and `release.yml` still +reference the deleted `gvr`/`noGvr` Gradle tasks (`assembleGvrDebug`, etc.) and will need +simplifying to a single variant before CI will pass again. + +## 2. `alvr_client_core.h` (provisional) + +This environment has no Rust toolchain or network access, so the real +`cargo xtask build-client-lib` -> `cbindgen` pipeline couldn't be run. The header at +`ALVR/build/alvr_client_core/alvr_client_core.h` was hand-derived by reading +`alvr/client_core/src/c_api.rs` and `alvr/common/src/c_api.rs` directly, following the project's +own `client_core/cbindgen.toml` (`QualifiedScreamingSnakeCase` enum tags, C-compatible layout). + +**Before shipping a real build:** run `prepare-alvr-deps.sh` (which invokes +`cargo xtask build-client-lib`) against your ALVR nightly checkout and diff the generated header +against this one. The Gradle `buildClientLib` task overwrites this same path on every build, so +this hand-written version is only there to make the port compile/review before that's possible. + +## 3. `alvr_main.cpp` API changes (v20 -> v21) + +| Area | v20 | v21 | +|---|---|---| +| View params | Baked into `alvr_send_tracking`'s 2nd arg, recomputed every tick | Sent once via `alvr_send_view_params()` (head-local pose only), whenever eye offsets/FOV change | +| Head prediction offset | `alvr_get_head_prediction_offset_ns()` | **Removed from the C API.** Now a fixed heuristic (`ESTIMATED_PREDICTION_OFFSET_NS`, 30ms) - see the `TODO(upstream)` comment. Consider upstreaming a C export for `ClientCoreContext::get_total_prediction_offset()`. | +| `alvr_resume_opengl` | 5 args (trailing `always_direct` bool) | 4 args, bool dropped | +| `alvr_render_lobby_opengl` | `AlvrViewInput[2]` | `AlvrLobbyViewParams[2]` (adds `swapchain_index` wrapper) + `render_background: bool` | +| `alvr_render_stream_opengl` | `(buffer, swapchain_indices[2])` | `(buffer, AlvrStreamViewParams[2], foveation_center_shifts)` - now requires per-eye `reprojection_rotation` + `fov`, computed via the new `alvr_rotation_delta()` helper. This is genuinely new capability: real head-rotation reprojection, which the Cardboard distortion renderer never had on its own. | +| `alvr_get_frame` | Returned `(view_params, buffer)` + timestamp | Returns `(timestamp, buffer)` only. View params/foveation centers for that frame are now fetched separately via `alvr_report_compositor_start(timestamp, &metadata)`, which can return `false` (no match) - the caller then reuses the last known-good metadata (implemented via `CTX.lastFrameMetadata`/`hasFrameMetadata`). | +| Foveation params | Parsed by hand from `alvr_get_settings_json()` | Delivered directly on `AlvrEvent::StreamingStarted` (`enable_foveated_encoding`, `foveated_encoding`) | +| `AlvrStreamConfig` | No upscaling fields | New `enable_upscaling`/`upscaling_edge_*`/`upscale_factor` fields (left disabled here) | +| `AlvrClientCapabilities` | - | New `max_view_width/height`, `prefer_10bit`, `preferred_encoding_gamma`, `prefer_hdr` (left at conservative defaults here) | +| Enum style | Flat C enums (`ALVR_EVENT_STREAMING_STARTED`, `ALVR_LOG_LEVEL_ERROR`) | **Unchanged** - still flat/unscoped despite `AlvrEvent` now being a genuine tagged union in Rust. Confirmed against the untouched `utils.h`, which already uses this convention. | + +Validated with a syntax-only compile (`g++ -fsyntax-only -Wall -Wextra`) against hand-written +stub headers for `jni.h`, `android/log.h`, `GLES3/gl3.h`, and `cardboard.h`, since the real +Android NDK/Cardboard SDK aren't available in this environment either. This caught two real bugs +before they'd have hit a real build: enums accidentally declared as scoped `enum class` instead of +plain C enums, and reversed pointer constness on `swapchain_textures`. It cannot substitute for +building against the real NDK + regenerated header, since the stubs are approximations. + +## Not done yet (separate, later step) + +ARCore integration and the rotational-drift complementary filter. `getPose()` in `alvr_main.cpp` +is explicitly commented as the extension point for that work - it currently returns Cardboard's +gyro-only orientation with position pinned to the origin (3DoF). diff --git a/code/mobile/android/PhoneVR/app/CMakeLists.txt b/code/mobile/android/PhoneVR/app/CMakeLists.txt index 358126a1..f50969b5 100644 --- a/code/mobile/android/PhoneVR/app/CMakeLists.txt +++ b/code/mobile/android/PhoneVR/app/CMakeLists.txt @@ -1,7 +1,5 @@ cmake_minimum_required(VERSION 3.4.3) -#use rtti, exceptions for asio library ## -std=c++17 -#set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -g -frtti -fexceptions -Wc++11-narrowing -Wreorder") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++17 -g") set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS} -O3 -std=c++17 -g") set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS} -O3 -std=c++17 -g") @@ -9,25 +7,14 @@ set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS} -O3 -std=c++17 -g") set(ANDROID_STL c++_static) set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY BOTH) -file(GLOB_RECURSE LIB_SRC - ../../../../common/libs/ifaddrs/*.c - ../../../../common/libs/json/single_include/*.hpp -) - add_library(native-lib-alvr SHARED src/main/cpp/alvr_main.cpp - ${LIB_SRC} ) set(libs_dir ${CMAKE_CURRENT_SOURCE_DIR}/libraries) set(alvr_build_dir ${CMAKE_CURRENT_SOURCE_DIR}/../ALVR/build/alvr_client_core) target_include_directories(native-lib-alvr - PUBLIC ../../../../common/libs/asio/asio/include - PUBLIC ../../../../common/libs/eigen/Eigen - PUBLIC ../../../../common/libs/ifaddrs - PUBLIC ../../../../common/libs/json/single_include/ - # ALVR Headers PUBLIC ${alvr_build_dir} PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/../cardboard @@ -38,7 +25,3 @@ target_link_libraries(native-lib-alvr ${alvr_build_dir}/${ANDROID_ABI}/libalvr_client_core.so ${libs_dir}/jni/${ANDROID_ABI}/libGfxPluginCardboard.so ) - -IF(${HAVEGVR}) -add_subdirectory(src/gvr) -ENDIF() diff --git a/code/mobile/android/PhoneVR/app/build.gradle b/code/mobile/android/PhoneVR/app/build.gradle index 399bcf30..b7ffa145 100644 --- a/code/mobile/android/PhoneVR/app/build.gradle +++ b/code/mobile/android/PhoneVR/app/build.gradle @@ -7,6 +7,9 @@ apply plugin: 'com.google.gms.google-services' apply from: '../versioning.gradle' apply plugin: "com.diffplug.spotless" +def arcore_libpath = "${rootProject.layout.buildDirectory.get()}/arcore-native" + +configurations { natives } def keystorePropertiesFile = rootProject.file("keystore.properties") def keystoreProperties = new Properties() keystoreProperties.load(new FileInputStream(keystorePropertiesFile)) @@ -41,6 +44,11 @@ android { applicationId "viritualisres.phonevr" minSdkVersion 24 targetSdkVersion 33 + + ndk { + abiFilters "arm64-v8a" + } + versionCode buildVersionCode() versionName buildVersionName() @@ -50,6 +58,8 @@ android { externalNativeBuild { cmake { cppFlags "-std=c++17 -fexceptions -frtti" //-DANDROID_USE_LEGACY_TOOLCHAIN_FILE=ON" + arguments "-DARCORE_LIBPATH=${arcore_libpath}/jni", + "-DARCORE_INCLUDE=${project.rootDir}/arcore-android-sdk-main/libraries/include" } } @@ -167,13 +177,16 @@ dependencies { gvrImplementation 'com.google.vr:sdk-base:1.200.0@aar' implementation 'com.google.cardboard.sdk:cardboard-sdk:1.17.0@aar' - implementation 'io.github.zxing-cpp:android:2.3.0-SNAPSHOT' + implementation 'io.github.zxing-cpp:android:2.3.0' implementation 'com.google.protobuf:protobuf-javalite:3.19.4' implementation 'androidx.camera:camera-core:1.3.4' implementation 'androidx.camera:camera-view:1.3.4' implementation 'androidx.camera:camera-lifecycle:1.3.4' implementation 'androidx.camera:camera-camera2:1.3.4' + implementation 'com.google.ar:core:1.46.0' + natives 'com.google.ar:core:1.46.0' + def acraVersion = '5.7.0' implementation "ch.acra:acra-mail:$acraVersion" implementation "ch.acra:acra-dialog:$acraVersion" @@ -199,6 +212,13 @@ task extractNdk(type: Copy) { include "jni/**/libgvr.so" } } + configurations.natives.files.each { f -> + copy { + from zipTree(f) + into arcore_libpath + include "jni/**/*" + } + } } task deleteNdk(type: Delete) { diff --git a/code/mobile/android/PhoneVR/app/src/main/AndroidManifest.xml b/code/mobile/android/PhoneVR/app/src/main/AndroidManifest.xml index c193327f..cec5caaf 100644 --- a/code/mobile/android/PhoneVR/app/src/main/AndroidManifest.xml +++ b/code/mobile/android/PhoneVR/app/src/main/AndroidManifest.xml @@ -12,6 +12,7 @@ + +#include #include #include +#include #include #include #include @@ -10,13 +12,74 @@ #include #include -#include "nlohmann/json.hpp" #include "utils.h" -using namespace nlohmann; + +#ifndef LOG +#define LOG(...) __android_log_print(ANDROID_LOG_INFO, "ALVR_EGLCHK", __VA_ARGS__) +#endif +#define EGLCHK(where) LOG("EGLCHK " where " ctx=%p surf=%p", eglGetCurrentContext(), eglGetCurrentSurface(EGL_DRAW)) + +struct EglStateGuard { + EGLDisplay dpy = eglGetCurrentDisplay(); + EGLContext ctx = eglGetCurrentContext(); + EGLSurface draw = eglGetCurrentSurface(EGL_DRAW); + EGLSurface read = eglGetCurrentSurface(EGL_READ); + ~EglStateGuard() { + if (ctx != EGL_NO_CONTEXT && eglGetCurrentContext() != ctx) + eglMakeCurrent(dpy, draw, read, ctx); + } +}; + +// TEMP DEBUG: burn texture names before creating the lobby swapchain, so the names handed to the +// ALVR library (17,18 instead of 1,2) can't coincide with names its own EGL context already uses. +// Remove once the context-sharing question is settled. +#define PVR_TEXTURE_NAME_TEST 1 + +// The ALVR graphics code aborts the process when it sees a GL error. glGetError() is sticky, so +// an error left behind by app/Cardboard code would be blamed on the library call. Drain (and +// log) any pending errors before each library call. +static void drainGlErrors(const char *where) { + GLenum e; + int n = 0; + while ((e = glGetError()) != GL_NO_ERROR && n++ < 16) + LOG("Pending GL error 0x%x before %s", e, where); +} + +// One-time GL driver info, to check feature support (external images) when debugging. +static void logGlInfo() { + const char *ver = (const char *) glGetString(GL_VERSION); + const char *ren = (const char *) glGetString(GL_RENDERER); + const char *ext = (const char *) glGetString(GL_EXTENSIONS); + LOG("GLINFO version=%s renderer=%s", ver ? ver : "?", ren ? ren : "?"); + LOG("GLINFO image_external_essl3=%d image_external=%d", + ext && strstr(ext, "GL_OES_EGL_image_external_essl3") != nullptr, + ext && strstr(ext, "GL_OES_EGL_image_external") != nullptr); +} + +// The header says later DecoderConfig events are ignored until reconnection. +static bool g_decoderCreated = false; + +// alvr_*_opengl() calls in this ALVR version can leave the thread with no current EGL context. +// Run them under a guard so the app's context/surface is rebound right after each call. +#define ALVR_GL_CALL(call) \ + do { \ + drainGlErrors(#call); \ + EglStateGuard _eglGuard; \ + call; \ + } while (0) + uint64_t HEAD_ID = alvr_path_string_to_id("/user/head"); +// v21 removed the alvr_get_head_prediction_offset_ns() C export (it's now +// ClientCoreContext::get_total_prediction_offset(), a Rust-only method used internally by +// client_openxr). There is currently no C-ABI replacement, so this is a fixed heuristic in the +// meantime. +// TODO(upstream): propose re-exposing get_total_prediction_offset() via c_api.rs so native +// clients don't have to guess this. +const uint64_t ESTIMATED_PREDICTION_OFFSET_NS = 30e6; + // Note: the Cardboard SDK cannot estimate display time and an heuristic is used instead. const uint64_t VSYNC_QUEUE_INTERVAL_NS = 50e6; const float FLOOR_HEIGHT = 1.5; @@ -40,19 +103,23 @@ struct NativeContext { bool streaming = false; std::thread inputThread; - // Une one texture per eye, no need for swapchains. + // Use one texture per eye, no need for swapchains. GLuint lobbyTextures[2] = {0, 0}; GLuint streamTextures[2] = {0, 0}; float eyeOffsets[2] = {0.0, 0.0}; AlvrFov fovArr[2] = {}; - AlvrViewParams viewParams[2] = {}; AlvrDeviceMotion deviceMotion = {}; + // Cache of the most recently matched frame metadata, since alvr_report_compositor_start() + // can legitimately return false (no match for this exact frame), in which case the caller + // is expected to reuse the previous metadata. + bool hasFrameMetadata = false; + AlvrVideoFrameMetadata lastFrameMetadata = {}; + NativeContext() { - memset(&fovArr, 0, (sizeof(fovArr)) / sizeof(int)); - memset(&viewParams, 0, (sizeof(viewParams)) / sizeof(int)); - memset(&deviceMotion, 0, (sizeof(deviceMotion)) / sizeof(int)); + memset(&fovArr, 0, sizeof(fovArr)); + memset(&deviceMotion, 0, sizeof(deviceMotion)); } }; @@ -105,6 +172,10 @@ AlvrFov getFov(CardboardEye eye) { return fov; } +// Head pose, in world/tracking space (not head-relative). Rotation currently comes straight from +// the Cardboard SDK's gyro-only sensor fusion, so it has no absolute position and will drift over +// time - this is the exact function that the planned ARCore + complementary filter work (a +// separate, later step) will extend to fuse in ARCore's 6DoF pose. AlvrPose getPose(uint64_t timestampNs) { AlvrPose pose = {}; @@ -115,47 +186,47 @@ AlvrPose getPose(uint64_t timestampNs) { auto inverseOrientation = AlvrQuat{q[0], q[1], q[2], q[3]}; pose.orientation = inverseQuat(inverseOrientation); + // Cardboard-only tracking has no position signal (3DoF), so position stays at the origin + // for now. + return pose; } -void updateViewConfigs(uint64_t targetTimestampNs = 0) { - if (!targetTimestampNs) - targetTimestampNs = GetBootTimeNano() + alvr_get_head_prediction_offset_ns(); - - AlvrPose headPose = getPose(targetTimestampNs); - - CTX.deviceMotion.device_id = HEAD_ID; - CTX.deviceMotion.pose = headPose; - - float headToEye[3] = {CTX.eyeOffsets[kLeft], 0.0, 0.0}; +// Local-space (head-at-origin) per-eye view parameters: just the eye offset and FOV. These rarely +// change (only when the lens distortion / viewer profile is recalculated), so this is sent once +// via alvr_send_view_params() rather than on every tracking update - unlike the v20 API, the +// moving head pose is no longer bundled into this call. +void sendViewParams() { + AlvrViewParams viewParams[2] = {}; - CTX.viewParams[kLeft].pose = headPose; - offsetPosWithQuat(headPose.orientation, headToEye, CTX.viewParams[kLeft].pose.position); - CTX.viewParams[kLeft].fov = CTX.fovArr[kLeft]; + for (int eye = 0; eye < 2; eye++) { + viewParams[eye].pose.orientation = AlvrQuat{0.0, 0.0, 0.0, 1.0}; + viewParams[eye].pose.position[0] = CTX.eyeOffsets[eye]; + viewParams[eye].pose.position[1] = 0.0; + viewParams[eye].pose.position[2] = 0.0; + viewParams[eye].fov = CTX.fovArr[eye]; + } - headToEye[0] = CTX.eyeOffsets[kRight]; - CTX.viewParams[kRight].pose = headPose; - offsetPosWithQuat(headPose.orientation, headToEye, CTX.viewParams[kRight].pose.position); - CTX.viewParams[kRight].fov = CTX.fovArr[kRight]; + alvr_send_view_params(viewParams); } void inputThread() { auto deadline = std::chrono::steady_clock::now(); - +ff info("inputThread: thread staring..."); while (CTX.streaming) { + auto targetTimestampNs = GetBootTimeNano() + ESTIMATED_PREDICTION_OFFSET_NS; - auto targetTimestampNs = GetBootTimeNano() + alvr_get_head_prediction_offset_ns(); - updateViewConfigs(targetTimestampNs); + CTX.deviceMotion.device_id = HEAD_ID; + CTX.deviceMotion.pose = getPose(targetTimestampNs); - alvr_send_tracking( - targetTimestampNs, CTX.viewParams, &CTX.deviceMotion, 1, nullptr, nullptr); + alvr_send_tracking(targetTimestampNs, &CTX.deviceMotion, 1, nullptr, nullptr);. deadline += std::chrono::nanoseconds((uint64_t) (1e9 / 60.f / 3)); std::this_thread::sleep_until(deadline); } } - +.. extern "C" JNIEXPORT jint JNI_OnLoad(JavaVM *vm, void *) { CTX.javaVm = vm; return JNI_VERSION_1_6; @@ -165,24 +236,31 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_initia JNIEnv *env, jobject obj, jint screenWidth, jint screenHeight, jfloat refreshRate) { CTX.javaContext = env->NewGlobalRef(obj); - uint32_t viewWidth = std::max(screenWidth, screenHeight) / 2; - uint32_t viewHeight = std::min(screenWidth, screenHeight); + uint32_t viewWidth = (std::max(screenWidth, screenHeight) / 2) & ~15u; // multiple of 16 + uint32_t viewHeight = std::min(screenWidth, screenHeight) & ~7u; // multiple of 8 + + // Route the Rust library's log output (including the reason for its aborts) to logcat. + alvr_initialize_logging(); alvr_initialize_android_context((void *) CTX.javaVm, (void *) CTX.javaContext); - float refreshRatesBuffer[1] = {refreshRate}; + static float refreshRatesBuffer[1]; + refreshRatesBuffer[0] = refreshRate; AlvrClientCapabilities caps = {}; - caps.default_view_height = viewHeight; caps.default_view_width = viewWidth; - caps.external_decoder = false; + caps.default_view_height = viewHeight; + caps.max_view_width = viewWidth; + caps.max_view_height = viewHeight; caps.refresh_rates = refreshRatesBuffer; caps.refresh_rates_count = 1; - caps.foveated_encoding = - true; // By default disable FFE (can be force-enabled by Server Settings + caps.foveated_encoding = false; caps.encoder_high_profile = true; - caps.encoder_10_bits = true; - caps.encoder_av1 = true; + caps.encoder_10_bits = false; + caps.encoder_av1 = false; + caps.prefer_10bit = false; + caps.preferred_encoding_gamma = 1.0; + caps.prefer_hdr = false; alvr_initialize(caps); @@ -218,8 +296,11 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_resume CardboardQrCode_destroy(buffer); CTX.running = true; - alvr_resume(); + + // at the start of renderNative, right after alvr_resume(...), and right before the final draw: + LOG("EGLCHK ctx=%p surf=%p", eglGetCurrentContext(), eglGetCurrentSurface(EGL_DRAW)); + EGLCHK("after alvr_resume()"); } extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_pauseNative(JNIEnv *, @@ -235,7 +316,8 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_pauseN extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_surfaceCreatedNative(JNIEnv *, jobject) { - alvr_initialize_opengl(); + logGlInfo(); + ALVR_GL_CALL(alvr_initialize_opengl()); CTX.glContextRecreated = true; } @@ -255,73 +337,167 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_sendBa extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_renderNative(JNIEnv *, jobject) { - try { - if (CTX.renderingParamsChanged) { - info("renderingParamsChanged, processing new params"); - uint8_t *buffer; - int size; - CardboardQrCode_getSavedDeviceParams(&buffer, &size); - - if (size == 0) { - return; - } + // at the start of renderNative, right after alvr_resume(...), and right before the final draw: + LOG("EGLCHK ctx=%p surf=%p", eglGetCurrentContext(), eglGetCurrentSurface(EGL_DRAW)); + EGLCHK("Start of renderNative()"); + + if (CTX.renderingParamsChanged) { + info("renderingParamsChanged, processing new params"); + uint8_t *buffer; + int size; + CardboardQrCode_getSavedDeviceParams(&buffer, &size); + + if (size == 0) { + return; + } - info("renderingParamsChanged, sending new params to alvr"); - if (CTX.lensDistortion) { - CardboardLensDistortion_destroy(CTX.lensDistortion); - CTX.lensDistortion = nullptr; - } - info("renderingParamsChanged, destroyed distortion"); - CTX.lensDistortion = - CardboardLensDistortion_create(buffer, size, CTX.screenWidth, CTX.screenHeight); + info("renderingParamsChanged, sending new params to alvr"); + if (CTX.lensDistortion) { + CardboardLensDistortion_destroy(CTX.lensDistortion); + CTX.lensDistortion = nullptr; + } + info("renderingParamsChanged, destroyed distortion"); + CTX.lensDistortion = + CardboardLensDistortion_create(buffer, size, CTX.screenWidth, CTX.screenHeight); - CardboardQrCode_destroy(buffer); - *buffer = 0; + CardboardQrCode_destroy(buffer); + *buffer = 0; - if (CTX.distortionRenderer) { - CardboardDistortionRenderer_destroy(CTX.distortionRenderer); - CTX.distortionRenderer = nullptr; - } - const CardboardOpenGlEsDistortionRendererConfig config{kGlTexture2D}; - CTX.distortionRenderer = CardboardOpenGlEs2DistortionRenderer_create(&config); - - for (int eye = 0; eye < 2; eye++) { - CardboardMesh mesh; - CardboardLensDistortion_getDistortionMesh( - CTX.lensDistortion, (CardboardEye) eye, &mesh); - CardboardDistortionRenderer_setMesh( - CTX.distortionRenderer, &mesh, (CardboardEye) eye); - - float matrix[16] = {}; - CardboardLensDistortion_getEyeFromHeadMatrix( - CTX.lensDistortion, (CardboardEye) eye, matrix); - CTX.eyeOffsets[eye] = matrix[12]; - } + if (CTX.distortionRenderer) { + CardboardDistortionRenderer_destroy(CTX.distortionRenderer); + CTX.distortionRenderer = nullptr; + } + const CardboardOpenGlEsDistortionRendererConfig config{kGlTexture2D}; + CTX.distortionRenderer = CardboardOpenGlEs2DistortionRenderer_create(&config); + + for (int eye = 0; eye < 2; eye++) { + CardboardMesh mesh; + CardboardLensDistortion_getDistortionMesh( + CTX.lensDistortion, (CardboardEye) eye, &mesh); + CardboardDistortionRenderer_setMesh( + CTX.distortionRenderer, &mesh, (CardboardEye) eye); + + float matrix[16] = {}; + CardboardLensDistortion_getEyeFromHeadMatrix( + CTX.lensDistortion, (CardboardEye) eye, matrix); + CTX.eyeOffsets[eye] = matrix[12]; + } - CTX.fovArr[kLeft] = getFov(kLeft); - CTX.fovArr[kRight] = getFov(kRight); + CTX.fovArr[kLeft] = getFov(kLeft); + CTX.fovArr[kRight] = getFov(kRight); - info("renderingParamsChanged, updating new view configs (FOV) to alvr"); - // alvr_send_views_config(fovArr, CTX.eyeOffsets[0] - CTX.eyeOffsets[1]); - } + info("renderingParamsChanged, sending new view configs (FOV) to alvr"); + sendViewParams(); + } - // Note: if GL context is recreated, old resources are already freed. - if (CTX.renderingParamsChanged && !CTX.glContextRecreated) { - info("Pausing ALVR since glContext is not recreated, deleting textures"); - alvr_pause_opengl(); + // Note: if GL context is recreated, old resources are already freed. + if (CTX.renderingParamsChanged && !CTX.glContextRecreated) { + info("Pausing ALVR since glContext is not recreated, deleting textures"); + ALVR_GL_CALL(alvr_pause_opengl()); + + GL(glDeleteTextures(2, CTX.lobbyTextures)); + } + + if (CTX.renderingParamsChanged || CTX.glContextRecreated) { + info("Rebuilding, binding textures, Resuming ALVR since glContextRecreated %b, " + "renderingParamsChanged %b", + CTX.renderingParamsChanged, + CTX.glContextRecreated); +#ifdef PVR_TEXTURE_NAME_TEST + { + static bool burned = false; + if (!burned) { + GLuint burn[16]; + glGenTextures(16, burn); + burned = true; + } + } +#endif + GL(glGenTextures(2, CTX.lobbyTextures)); + + for (auto &lobbyTexture : CTX.lobbyTextures) { + GL(glBindTexture(GL_TEXTURE_2D, lobbyTexture)); + GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE)); + GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE)); + GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR)); + GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR)); + GL(glTexImage2D(GL_TEXTURE_2D, + 0, + GL_RGB, + CTX.screenWidth / 2, + CTX.screenHeight, + 0, + GL_RGB, + GL_UNSIGNED_BYTE, + nullptr)); + } - GL(glDeleteTextures(2, CTX.lobbyTextures)); + // The ALVR library renders in its own EGL context (now a share context of this one). + // Object creation/changes in this context are only guaranteed visible to the other + // context after a flush/finish, so finish before handing the texture names over. + glFinish(); + LOG("LOBBYTEX names=%u,%u isTexture=%d,%d size=%dx%d", + (unsigned) CTX.lobbyTextures[0], + (unsigned) CTX.lobbyTextures[1], + (int) glIsTexture(CTX.lobbyTextures[0]), + (int) glIsTexture(CTX.lobbyTextures[1]), + CTX.screenWidth / 2, + CTX.screenHeight); + + const uint32_t *targetViews[2] = {(const uint32_t *) &CTX.lobbyTextures[0], + (const uint32_t *) &CTX.lobbyTextures[1]}; + // v21 dropped the trailing bool (was "always direct") - resume takes 4 args now. + // alvr_resume_opengl() leaves the thread with no current EGL context (observed: + // ctx/surf go to 0x0), which makes every later GL call fail and GLSurfaceView stop + // rendering. The guard captures the app's EGL state before the call and rebinds it + // immediately afterwards. + EGLCHK("before alvr_resume_opengl"); + { + EglStateGuard guard; + alvr_resume_opengl(CTX.screenWidth / 2, CTX.screenHeight, targetViews, 1); } + EGLCHK("after alvr_resume_opengl (restored)"); + + CTX.renderingParamsChanged = false; + CTX.glContextRecreated = false; + } - if (CTX.renderingParamsChanged || CTX.glContextRecreated) { - info("Rebuilding, binding textures, Resuming ALVR since glContextRecreated %b, " - "renderingParamsChanged %b", - CTX.renderingParamsChanged, - CTX.glContextRecreated); - GL(glGenTextures(2, CTX.lobbyTextures)); + AlvrEvent event; + while (alvr_poll_event(&event)) { + if (event.tag == ALVR_EVENT_HUD_MESSAGE_UPDATED) { + auto message_length = alvr_hud_message(nullptr); + auto message_buffer = std::vector(message_length); - for (auto &lobbyTexture : CTX.lobbyTextures) { - GL(glBindTexture(GL_TEXTURE_2D, lobbyTexture)); + alvr_hud_message(&message_buffer[0]); + info("ALVR Poll Event: HUD Message Update - %s", &message_buffer[0]); + + // 3) inside the HUD branch, right after the call + if (message_length > 0) + ALVR_GL_CALL(alvr_update_hud_message_opengl(&message_buffer[0])); + EGLCHK("after update_hud_message"); + } + if (event.tag == ALVR_EVENT_STREAMING_STARTED) { + info("ALVR Poll Event: ALVR_EVENT_STREAMING_STARTED, generating and binding " + "textures..."); + auto &started = event.STREAMING_STARTED; + + info("Streaming started: %dx%d, refresh hint=%f, gamma=%f, hdr=%b, ffe=%b", + started.view_width, + started.view_height, + started.refresh_rate_hint, + started.encoding_gamma, + started.enable_hdr, + started.enable_foveated_encoding); + + // v21 delivers the negotiated foveation params directly in the event - no more + // manual settings-JSON parsing needed here (settings.video.foveated_encoding, as + // the v20 code did). + CTX.hasFrameMetadata = false; + + GL(glGenTextures(2, CTX.streamTextures)); + + for (auto &streamTexture : CTX.streamTextures) { + GL(glBindTexture(GL_TEXTURE_2D, streamTexture)); GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE)); GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE)); GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR)); @@ -329,212 +505,230 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render GL(glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, - CTX.screenWidth / 2, - CTX.screenHeight, + started.view_width, + started.view_height, 0, GL_RGB, GL_UNSIGNED_BYTE, nullptr)); } - const uint32_t *targetViews[2] = {(uint32_t *) &CTX.lobbyTextures[0], - (uint32_t *) &CTX.lobbyTextures[1]}; - alvr_resume_opengl(CTX.screenWidth / 2, CTX.screenHeight, targetViews, 1, true); - - CTX.renderingParamsChanged = false; - CTX.glContextRecreated = false; - } - - AlvrEvent event; - while (alvr_poll_event(&event)) { - if (event.tag == ALVR_EVENT_HUD_MESSAGE_UPDATED) { - auto message_length = alvr_hud_message(nullptr); - auto message_buffer = std::vector(message_length); + CTX.fovArr[0] = getFov((CardboardEye) 0); + CTX.fovArr[1] = getFov((CardboardEye) 1); + sendViewParams(); + + info("ALVR Poll Event: ALVR_EVENT_STREAMING_STARTED, View configs updated..."); + + auto leftIntHandle = (uint32_t) CTX.streamTextures[0]; + auto rightIntHandle = (uint32_t) CTX.streamTextures[1]; + const uint32_t *textureHandles[2] = {&leftIntHandle, &rightIntHandle}; + + auto render_config = AlvrStreamConfig{}; + render_config.view_resolution_width = started.view_width; + render_config.view_resolution_height = started.view_height; + render_config.swapchain_textures = textureHandles; + render_config.swapchain_length = 1; + + render_config.enable_foveation = started.enable_foveated_encoding; + render_config.foveated_encoding = started.foveated_encoding; + + // Not opting into server-side upscaling yet. + render_config.enable_upscaling = false; + + ALVR_GL_CALL(alvr_start_stream_opengl(render_config)); + + info("ALVR Poll Event: ALVR_EVENT_STREAMING_STARTED, opengl stream started and " + "input Thread started..."); + CTX.streaming = true; + CTX.inputThread = std::thread(inputThread); + } else if (event.tag == ALVR_EVENT_DECODER_CONFIG) { + + /* + * v21 no longer sends the decoder config in the ALVR_EVENT_STREAMING_STARTED event, + * so we have to create the decoder here instead. The decoder config is sent in a + * separate event, ALVR_EVENT_DECODER_CONFIG, which is guaranteed to arrive before + * the first frame. + */ + /* + if (!g_decoderCreated) { + auto configSize = alvr_get_decoder_config(nullptr); + auto configBuffer = std::vector(configSize); + alvr_get_decoder_config(configBuffer.data()); + + AlvrDecoderConfig decoderConfig = {}; + decoderConfig.codec = event.DECODER_CONFIG.codec; + decoderConfig.force_software_decoder = true; + decoderConfig.max_buffering_frames = 1.5f; + decoderConfig.buffering_history_weight = 0.9f; + decoderConfig.options = nullptr; + decoderConfig.options_count = 0; + decoderConfig.config_buffer = + reinterpret_cast(configBuffer.data()); + decoderConfig.config_buffer_size = configSize; + + info("ALVR Poll Event: ALVR_EVENT_DECODER_CONFIG, codec=%d, config bytes=%llu", + (int) decoderConfig.codec, (unsigned long long) configSize); + alvr_create_decoder(decoderConfig); + g_decoderCreated = true; + } + */ + + if (!g_decoderCreated) { + auto configSize = alvr_get_decoder_config(nullptr); + auto configBuffer = std::vector(configSize); + alvr_get_decoder_config(configBuffer.data()); + + // false = try the phone's hardware decoder; flip to true if it fails again. + constexpr bool FORCE_SOFTWARE_DECODER = false; + + // Same options the stock ALVR client gets from the server's defaults. + AlvrMediacodecOption decoderOptions[3] = {}; + decoderOptions[0].key = "operating-rate"; + decoderOptions[0].ty = ALVR_MEDIACODEC_PROP_TYPE_INT32; + decoderOptions[0].value.int32 = INT32_MAX; + decoderOptions[1].key = "priority"; + decoderOptions[1].ty = ALVR_MEDIACODEC_PROP_TYPE_INT32; + decoderOptions[1].value.int32 = 0; + decoderOptions[2].key = "vendor.qti-ext-dec-low-latency.enable"; + decoderOptions[2].ty = ALVR_MEDIACODEC_PROP_TYPE_INT32; + decoderOptions[2].value.int32 = 1; + + AlvrDecoderConfig decoderConfig = {}; + decoderConfig.codec = event.DECODER_CONFIG.codec; + decoderConfig.force_software_decoder = FORCE_SOFTWARE_DECODER; + decoderConfig.max_buffering_frames = 1.5f; + decoderConfig.buffering_history_weight = 0.9f; + decoderConfig.options = decoderOptions; + decoderConfig.options_count = 3; + decoderConfig.config_buffer = + reinterpret_cast(configBuffer.data()); + decoderConfig.config_buffer_size = configSize; + + info("ALVR Poll Event: ALVR_EVENT_DECODER_CONFIG, codec=%d, config bytes=%llu, " + "software=%d", + (int) decoderConfig.codec, (unsigned long long) configSize, + (int) FORCE_SOFTWARE_DECODER); + alvr_create_decoder(decoderConfig); + g_decoderCreated = true; + } - alvr_hud_message(&message_buffer[0]); - info("ALVR Poll Event: HUD Message Update - %s", &message_buffer[0]); + } else if (event.tag == ALVR_EVENT_STREAMING_STOPPED) { + info("ALVR Poll Event: ALVR_EVENT_STREAMING_STOPPED, Waiting for inputThread to " + "join..."); + CTX.streaming = false; + CTX.inputThread.join(); - if (message_length > 0) - alvr_update_hud_message_opengl(&message_buffer[0]); - } - if (event.tag == ALVR_EVENT_STREAMING_STARTED) { - info("ALVR Poll Event: ALVR_EVENT_STREAMING_STARTED, generating and binding " - "textures..."); - auto config = event.STREAMING_STARTED; - - auto settings_len = alvr_get_settings_json(nullptr); - auto settings_buffer = std::vector(settings_len); - alvr_get_settings_json(&settings_buffer[0]); - - info("Got settings from ALVR Server - %s", &settings_buffer[0]); - if (settings_len > 900) // to workthough logcat buffer limit - info("Got settings from ALVR Server - %s", &settings_buffer[900]); - json settings_json = json::parse(&settings_buffer[0]); - - GL(glGenTextures(2, CTX.streamTextures)); - - for (auto &streamTexture : CTX.streamTextures) { - GL(glBindTexture(GL_TEXTURE_2D, streamTexture)); - GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE)); - GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE)); - GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR)); - GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR)); - GL(glTexImage2D(GL_TEXTURE_2D, - 0, - GL_RGB, - config.view_width, - config.view_height, - 0, - GL_RGB, - GL_UNSIGNED_BYTE, - nullptr)); - } - - CTX.fovArr[0] = getFov((CardboardEye) 0); - CTX.fovArr[1] = getFov((CardboardEye) 1); - - info("ALVR Poll Event: ALVR_EVENT_STREAMING_STARTED, View configs updated..."); - - auto leftIntHandle = (uint32_t) CTX.streamTextures[0]; - auto rightIntHandle = (uint32_t) CTX.streamTextures[1]; - const uint32_t *textureHandles[2] = {&leftIntHandle, &rightIntHandle}; - - auto render_config = AlvrStreamConfig{}; - render_config.view_resolution_width = config.view_width; - render_config.view_resolution_height = config.view_height; - render_config.swapchain_textures = textureHandles; - render_config.swapchain_length = 1; - - render_config.enable_foveation = false; - if (!settings_json["video"].is_null()) { - if (!settings_json["video"]["foveated_encoding"].is_null()) { - info("settings_json.video.foveated_encoding is %s", - settings_json["video"]["foveated_encoding"].dump().c_str()); - - // Foveated encoding would be a "Enabled": {Array} or "Disabled" String - if (!settings_json["video"]["foveated_encoding"].is_string()) { - render_config.enable_foveation = true; - render_config.foveation_center_size_x = - settings_json["video"]["foveated_encoding"]["Enabled"] - ["center_size_x"]; - render_config.foveation_center_size_y = - settings_json["video"]["foveated_encoding"]["Enabled"] - ["center_size_y"]; - render_config.foveation_center_shift_x = - settings_json["video"]["foveated_encoding"]["Enabled"] - ["center_shift_x"]; - render_config.foveation_center_shift_y = - settings_json["video"]["foveated_encoding"]["Enabled"] - ["center_shift_y"]; - render_config.foveation_edge_ratio_x = - settings_json["video"]["foveated_encoding"]["Enabled"] - ["edge_ratio_x"]; - render_config.foveation_edge_ratio_y = - settings_json["video"]["foveated_encoding"]["Enabled"] - ["edge_ratio_y"]; - } else - info("foveated_encoding is Disabled"); - } else - error("settings_json doesn't have a video.foveated_encoding key"); - } else - error("settings_json doesn't have a video key"); - - info("Settings for foveation:"); - info("render_config.enable_foveation: %b", render_config.enable_foveation); - info("render_config.foveation_center_size_x: %f", - render_config.foveation_center_size_x); - info("render_config.foveation_center_size_y: %f", - render_config.foveation_center_size_y); - info("render_config.foveation_center_shift_x: %f", - render_config.foveation_center_shift_x); - info("render_config.foveation_center_shift_y: %f", - render_config.foveation_center_shift_y); - info("render_config.foveation_edge_ratio_x: %f", - render_config.foveation_edge_ratio_x); - info("render_config.foveation_edge_ratio_y: %f", - render_config.foveation_edge_ratio_y); - - alvr_start_stream_opengl(render_config); - - info("ALVR Poll Event: ALVR_EVENT_STREAMING_STARTED, opengl stream started and " - "input " - "Thread started..."); - CTX.streaming = true; - CTX.inputThread = std::thread(inputThread); - - } else if (event.tag == ALVR_EVENT_STREAMING_STOPPED) { - info("ALVR Poll Event: ALVR_EVENT_STREAMING_STOPPED, Waiting for inputThread to " - "join..."); - CTX.streaming = false; - CTX.inputThread.join(); - - GL(glDeleteTextures(2, CTX.streamTextures)); - info("ALVR Poll Event: ALVR_EVENT_STREAMING_STOPPED, Stream stopped deleted " - "textures."); + if (g_decoderCreated) { + alvr_destroy_decoder(); + g_decoderCreated = false; } - } - CardboardEyeTextureDescription viewsDescs[2] = {}; - for (auto &viewsDesc : viewsDescs) { - viewsDesc.left_u = 0.0; - viewsDesc.right_u = 1.0; - viewsDesc.top_v = 1.0; - viewsDesc.bottom_v = 0.0; + GL(glDeleteTextures(2, CTX.streamTextures)); + info("ALVR Poll Event: ALVR_EVENT_STREAMING_STOPPED, Stream stopped deleted " + "textures."); } + } + // 4) right after the closing brace of the `while (alvr_poll_event(&event))` loop, + // before `CardboardEyeTextureDescription viewsDescs[2] = {};` + EGLCHK("after event loop"); + + CardboardEyeTextureDescription viewsDescs[2] = {}; + for (auto &viewsDesc : viewsDescs) { + viewsDesc.left_u = 0.0; + viewsDesc.right_u = 1.0; + viewsDesc.top_v = 1.0; + viewsDesc.bottom_v = 0.0; + } + + if (CTX.streaming) { + void *streamHardwareBuffer = nullptr; + uint64_t timestampNs = 0; - if (CTX.streaming) { - void *streamHardwareBuffer = nullptr; + if (!alvr_get_frame(×tampNs, &streamHardwareBuffer)) { + return; + } - AlvrViewParams dummyViewParams; - auto timestampNs = alvr_get_frame(&dummyViewParams, &streamHardwareBuffer); + // v21 no longer returns view params from alvr_get_frame() directly; fetch the + // server-aligned metadata (view params + foveation centers) matched to this frame here + // instead. If there's no match for this exact frame, fall back to the last known-good + // metadata, per the API doc for this function. + AlvrVideoFrameMetadata metadata; + if (alvr_report_compositor_start(timestampNs, &metadata)) { + CTX.lastFrameMetadata = metadata; + CTX.hasFrameMetadata = true; + } else if (CTX.hasFrameMetadata) { + metadata = CTX.lastFrameMetadata; + } else { + // No metadata at all yet (first frame) - nothing sensible to reproject against. + return; + } - if (timestampNs == -1) { - return; - } + // Current best head-orientation estimate, for reprojecting away the latency between + // when the server rendered this frame and now. The Cardboard SDK's own distortion + // renderer has no reprojection support of its own; this reprojection_rotation is what + // lets v21 do it properly even on this simple client. + AlvrPose currentPose = getPose(GetBootTimeNano() + VSYNC_QUEUE_INTERVAL_NS); + + AlvrStreamViewParams streamViewParams[2] = {}; + for (int eye = 0; eye < 2; eye++) { + streamViewParams[eye].swapchain_index = 0; + streamViewParams[eye].reprojection_rotation = + alvr_rotation_delta(metadata.view_params[eye].pose.orientation, + currentPose.orientation); + streamViewParams[eye].fov = metadata.view_params[eye].fov; + } - uint32_t swapchainIndices[2] = {0, 0}; - alvr_render_stream_opengl(streamHardwareBuffer, swapchainIndices); + const float(*foveationCenters)[2][2] = + metadata.has_foveation_centers ? &metadata.foveation_center_shifts : nullptr; - alvr_report_submit(timestampNs, 0); + EGLCHK("before alvr_render_stream_opengl()"); + ALVR_GL_CALL( + alvr_render_stream_opengl(streamHardwareBuffer, streamViewParams, foveationCenters)); + EGLCHK("after alvr_render_stream_opengl()"); - viewsDescs[0].texture = CTX.streamTextures[0]; - viewsDescs[1].texture = CTX.streamTextures[1]; - } else { - AlvrPose pose = getPose(GetBootTimeNano() + VSYNC_QUEUE_INTERVAL_NS); + alvr_report_submit(timestampNs, 0); - AlvrViewInput viewInputs[2] = {}; - for (int eye = 0; eye < 2; eye++) { - float headToEye[3] = {CTX.eyeOffsets[eye], 0.0, 0.0}; - // offset head pos to Eye Position - offsetPosWithQuat(pose.orientation, headToEye, viewInputs[eye].pose.position); + viewsDescs[0].texture = CTX.streamTextures[0]; + viewsDescs[1].texture = CTX.streamTextures[1]; + } else { + AlvrPose pose = getPose(GetBootTimeNano() + VSYNC_QUEUE_INTERVAL_NS); - viewInputs[eye].pose.orientation = pose.orientation; - viewInputs[eye].fov = getFov((CardboardEye) eye); - viewInputs[eye].swapchain_index = 0; - } - alvr_render_lobby_opengl(viewInputs); + AlvrLobbyViewParams viewInputs[2] = {}; + for (int eye = 0; eye < 2; eye++) { + float headToEye[3] = {CTX.eyeOffsets[eye], 0.0, 0.0}; + // offset head pos to Eye Position + offsetPosWithQuat( + pose.orientation, headToEye, viewInputs[eye].view_params.pose.position); - viewsDescs[0].texture = CTX.lobbyTextures[0]; - viewsDescs[1].texture = CTX.lobbyTextures[1]; + viewInputs[eye].view_params.pose.orientation = pose.orientation; + viewInputs[eye].view_params.fov = getFov((CardboardEye) eye); + viewInputs[eye].swapchain_index = 0; + } + static int lobbyTexLogs = 0; + if (lobbyTexLogs++ < 3) { + LOG("LOBBYTEX before lobby render: names=%u,%u isTexture=%d,%d", + (unsigned) CTX.lobbyTextures[0], + (unsigned) CTX.lobbyTextures[1], + (int) glIsTexture(CTX.lobbyTextures[0]), + (int) glIsTexture(CTX.lobbyTextures[1])); } + EGLCHK("before alvr_render_lobby_opengl()"); + ALVR_GL_CALL(alvr_render_lobby_opengl(viewInputs, /*render_background=*/true)); + EGLCHK("after alvr_render_lobby_opengl()"); - // Note: the Cardboard SDK does not support reprojection! - // todo: manually implement it? - - // info("nativeRendered: Rendering to Display..."); - CardboardDistortionRenderer_renderEyeToDisplay(CTX.distortionRenderer, - 0, - 0, - 0, - CTX.screenWidth, - CTX.screenHeight, - &viewsDescs[0], - &viewsDescs[1]); - } catch (const json::exception &e) { - error(std::string(std::string(__FUNCTION__) + std::string(__FILE_NAME__) + - std::string(e.what())) - .c_str()); + viewsDescs[0].texture = CTX.lobbyTextures[0]; + viewsDescs[1].texture = CTX.lobbyTextures[1]; } + EGLCHK("before CardboardDistortionRenderer_renderEyeToDisplay()"); + CardboardDistortionRenderer_renderEyeToDisplay(CTX.distortionRenderer, + 0, + 0, + 0, + CTX.screenWidth, + CTX.screenHeight, + &viewsDescs[0], + &viewsDescs[1]); + EGLCHK("after CardboardDistortionRenderer_renderEyeToDisplay()"); } extern "C" JNIEXPORT void JNICALL diff --git a/code/mobile/android/PhoneVR/app/src/main/res/values/strings.xml b/code/mobile/android/PhoneVR/app/src/main/res/values/strings.xml index ce5664f9..0e02296b 100644 --- a/code/mobile/android/PhoneVR/app/src/main/res/values/strings.xml +++ b/code/mobile/android/PhoneVR/app/src/main/res/values/strings.xml @@ -1,6 +1,7 @@ PhoneVR PhoneVR noGvr + PhoneVR Oh no :( PVR has unfortunately crashed. Dev team will be notified. You can create an issue on Github or Discord. PC IP Auto diff --git a/code/mobile/android/PhoneVR/prepare-alvr-deps.sh b/code/mobile/android/PhoneVR/prepare-alvr-deps.sh old mode 100644 new mode 100755 index 524bb187..d1df3391 --- a/code/mobile/android/PhoneVR/prepare-alvr-deps.sh +++ b/code/mobile/android/PhoneVR/prepare-alvr-deps.sh @@ -1,8 +1,3 @@ -if [ "$1" == "nogvr" ]; then -nogvr=true -shift -fi - rustup target add \ aarch64-linux-android \ armv7-linux-androideabi \ @@ -44,10 +39,3 @@ mv "${CARB_REPO_NAME}/sdk/build/outputs/aar/sdk-release.aar" cardboard/cardboard cp "${CARB_REPO_NAME}/sdk/include/cardboard.h" cardboard/cardboard.h rm -r "${CARB_REPO_NAME}" -rm -r "gvr-android-sdk-1.200" -if [ ! $nogvr ]; then -curl -sLS "https://github.com/googlevr/gvr-android-sdk/releases/download/v1.200/gvr-android-sdk-1.200.zip" > download.zip -unzip download.zip -rm download.zip -fi - From 057113cd943f352cb465bb693c5b12a38187cde8 Mon Sep 17 00:00:00 2001 From: Rob Oates Date: Mon, 5 Oct 2026 01:33:39 +0900 Subject: [PATCH 02/29] Add wgpu-hal patch: share the app's EGL context with ALVR's GL context --- .../wgpu-hal-25.0.2-share-egl-context.patch | 99 +++++++++++++++++++ 1 file changed, 99 insertions(+) create mode 100644 code/mobile/android/PhoneVR/patches/wgpu-hal-25.0.2-share-egl-context.patch diff --git a/code/mobile/android/PhoneVR/patches/wgpu-hal-25.0.2-share-egl-context.patch b/code/mobile/android/PhoneVR/patches/wgpu-hal-25.0.2-share-egl-context.patch new file mode 100644 index 00000000..ffb250f9 --- /dev/null +++ b/code/mobile/android/PhoneVR/patches/wgpu-hal-25.0.2-share-egl-context.patch @@ -0,0 +1,99 @@ +diff -ruN -x '*.bak' -x target a/.cargo-ok b/.cargo-ok +--- a/.cargo-ok 1970-01-01 09:00:00.000000000 +0900 ++++ b/.cargo-ok 2026-10-05 01:32:32.592471619 +0900 +@@ -0,0 +1 @@ ++{"v":1} +\ No newline at end of file +diff -ruN -x '*.bak' -x target a/src/gles/egl.rs b/src/gles/egl.rs +--- a/src/gles/egl.rs 2026-10-05 01:32:32.518542445 +0900 ++++ b/src/gles/egl.rs 2026-10-05 01:32:32.561883826 +0900 +@@ -410,9 +410,13 @@ + impl<'a> Drop for AdapterContextLock<'a> { + fn drop(&mut self) { + if let Some(egl) = self.egl.take() { ++ log::info!("ALVR GLES UNLOCK: releasing wgpu EGL context"); ++ + egl.instance + .make_current(egl.display, None, None, None) + .unwrap(); ++ ++ log::info!("ALVR GLES UNLOCK: wgpu EGL context released"); + } + } + } +@@ -449,7 +453,9 @@ + .expect("Could not lock adapter context. This is most-likely a deadlock."); + + let egl = self.egl.as_ref().map(|egl| { ++ log::info!("ALVR GLES LOCK: making wgpu EGL context current"); + egl.make_current(); ++ log::info!("ALVR GLES LOCK: wgpu EGL context is current"); + EglContextLock { + instance: &egl.instance, + display: egl.display, +@@ -654,11 +660,16 @@ + gl_context_attributes.extend(&context_attributes); + gles_context_attributes.extend(&context_attributes); + +- let context = if supports_opengl { +- egl.create_context(display, config, None, &gl_context_attributes) ++ // ALVR: share GL objects with the app's current EGL context, so textures created by the ++ // app (glGenTextures) are visible inside this context. ++ let share_context = egl.get_current_context(); ++ log::info!("ALVR GLES: share context = {:?}", share_context.map(|c| c.as_ptr())); ++ ++ let context = if supports_opengl { ++ egl.create_context(display, config, share_context, &gl_context_attributes) + .or_else(|_| { + egl.bind_api(khronos_egl::OPENGL_ES_API).unwrap(); +- egl.create_context(display, config, None, &gles_context_attributes) ++ egl.create_context(display, config, share_context, &gles_context_attributes) + }) + .map_err(|e| { + crate::InstanceError::with_source( +@@ -667,7 +678,7 @@ + ) + }) + } else { +- egl.create_context(display, config, None, &gles_context_attributes) ++ egl.create_context(display, config, share_context, &gles_context_attributes) + .map_err(|e| { + crate::InstanceError::with_source( + String::from("unable to create GLES 3.x context"), +diff -ruN -x '*.bak' -x target a/src/gles/queue.rs b/src/gles/queue.rs +--- a/src/gles/queue.rs 2026-10-05 01:32:32.518643908 +0900 ++++ b/src/gles/queue.rs 2026-10-05 01:32:32.568580204 +0900 +@@ -144,7 +144,14 @@ + get_2d_target(target, view.array_layers.start), + Some(raw), + view.mip_levels.start as i32, +- ) ++ ); ++ log::info!( ++ "ALVR GLES ATTACH: texture={} attachment=0x{:X} error=0x{:X} framebuffer=0x{:X}", ++ raw.0, ++ attachment, ++ gl.get_error(), ++ gl.check_framebuffer_status(glow::DRAW_FRAMEBUFFER), ++ ); + }; + } + } +@@ -1818,8 +1825,17 @@ + _surface_textures: &[&super::Texture], + (signal_fence, signal_value): (&mut super::Fence, crate::FenceValue), + ) -> Result<(), crate::DeviceError> { ++ log::info!( ++ "ALVR GLES SUBMIT: entered submit(), command_buffers={}", ++ command_buffers.len() ++ ); ++ + let shared = Arc::clone(&self.shared); ++ ++ log::info!("ALVR GLES SUBMIT: about to lock EGL context"); + let gl = &shared.context.lock(); ++ log::info!("ALVR GLES SUBMIT: EGL context locked"); ++ + for cmd_buf in command_buffers.iter() { + // The command encoder assumes a default state when encoding the command buffer. + // Always reset the state between command_buffers to reflect this assumption. Do From 478c077aadb8160d1fde6d3c9869e84f8239f494 Mon Sep 17 00:00:00 2001 From: Rob Oates Date: Mon, 5 Oct 2026 01:41:00 +0900 Subject: [PATCH 03/29] Update MIGRATION_NOTES --- MIGRATION_NOTES.md | 185 ++++++++++++++++++++++++++++++--------------- 1 file changed, 123 insertions(+), 62 deletions(-) diff --git a/MIGRATION_NOTES.md b/MIGRATION_NOTES.md index 1aad3f4d..36383d51 100644 --- a/MIGRATION_NOTES.md +++ b/MIGRATION_NOTES.md @@ -1,69 +1,130 @@ # PhoneVR: ALVR v20 -> v21 migration notes -Generated while porting the Android client's `main` flavor (the one built on `alvr_client_core`, -not the legacy `gvr` custom-protocol flavor) from ALVR v20.8.0 to the v21 nightly -(commit `dc84d6835d4125226f9698f1f052629852d86e31`). - -## 1. Pruned (removed) - -- `code/mobile/android/PhoneVR/app/src/gvr/` and `app/src/androidTestGvr/` - the legacy, - fully custom UDP-socket protocol client (`PVRSockets`/`PVRRenderer`, Google VR SDK). Superseded - entirely by the `main` flavor, which already links `alvr_client_core` directly. -- `code/mobile/mobile-common/` - only consumed by the removed `gvr` flavor. -- `gvrImplementation 'com.google.vr:sdk-base:...'` and the GVR SDK unzip step in - `extractNdk` (`app/build.gradle`). -- The `flavorDimensions "gvr"` / `productFlavors { gvr / noGvr }` split - now a single - build variant. -- `IF(${HAVEGVR}) add_subdirectory(src/gvr) ENDIF()` in `CMakeLists.txt`, plus the now-unused - `asio`/`eigen`/`ifaddrs` include dirs (only needed by the removed custom protocol). -- The GVR SDK download block in `prepare-alvr-deps.sh`, and the `nogvr` CLI flag it existed for. -- Manual settings-JSON parsing for foveated encoding in `alvr_main.cpp` - v21 delivers - `foveated_encoding` directly on the `StreamingStarted` event, so the `nlohmann/json` dependency - is gone from this file entirely. - -**Not yet touched, but likely also obsolete now:** `driver/PVRServer` (a SteamVR driver for the -removed custom protocol's PC side) and `code/windows`. Left alone since they're outside the phone -client and weren't explicitly in scope. `.github/workflows/build.yml` and `release.yml` still -reference the deleted `gvr`/`noGvr` Gradle tasks (`assembleGvrDebug`, etc.) and will need -simplifying to a single variant before CI will pass again. - -## 2. `alvr_client_core.h` (provisional) - -This environment has no Rust toolchain or network access, so the real -`cargo xtask build-client-lib` -> `cbindgen` pipeline couldn't be run. The header at -`ALVR/build/alvr_client_core/alvr_client_core.h` was hand-derived by reading -`alvr/client_core/src/c_api.rs` and `alvr/common/src/c_api.rs` directly, following the project's -own `client_core/cbindgen.toml` (`QualifiedScreamingSnakeCase` enum tags, C-compatible layout). - -**Before shipping a real build:** run `prepare-alvr-deps.sh` (which invokes -`cargo xtask build-client-lib`) against your ALVR nightly checkout and diff the generated header -against this one. The Gradle `buildClientLib` task overwrites this same path on every build, so -this hand-written version is only there to make the port compile/review before that's possible. +Last updated: 2026-10-05 + +## Status + +The Android client (the `main` flavor, built on `alvr_client_core`) now connects to an ALVR v21 +streamer, decodes video on the phone's hardware decoder (H.264 and HEVC verified) and renders it +through the Cardboard distortion renderer. Head orientation comes from the Cardboard head tracker +(3DoF). Position is not tracked yet. + +- **ALVR version:** v21 is not a tagged release. It is the `master` branch of `alvr-org/ALVR` + (the main repo's newest tag is still v20.14.1). This port was built against commit `9f118394` + (`21.0.0-dev14`). The client library header from `dc84d683` (also `21.0.0-dev14`) is identical. +- **Test device:** an Android phone with an Adreno 619 GPU, OpenGL ES 3.2, with + `GL_OES_EGL_image_external` supported. +- **Build host:** Ubuntu in VirtualBox, JDK 17, Rust 1.97.1, Android NDK 25.2.9519653. + +## 1. Build configuration changes + +No source folders were removed in this branch. `app/src/gvr/`, `app/src/androidTestGvr/` and +`code/mobile/mobile-common/` are still present. They belong to the legacy custom-protocol client +and should no longer be built, because the build files below were changed to produce only the +ALVR-based variant. They can be deleted in a later cleanup. + +The build files were changed to produce a single ALVR-based variant (from the original port notes): + +- `app/build.gradle`: the `gvr` / `noGvr` product flavor split and the Google VR SDK dependency + and unzip step were removed. +- `app/CMakeLists.txt`: the `src/gvr` subdirectory and the include directories only the legacy + custom protocol needed (`asio`, `eigen`, `ifaddrs`) were removed. +- `prepare-alvr-deps.sh`: the GVR SDK download and the `nogvr` flag were removed. +- `alvr_main.cpp`: foveated-encoding settings are no longer parsed from the settings JSON, because + v21 delivers them on the `StreamingStarted` event. The `nlohmann/json` dependency is gone from + this file. + +Consequence: this branch no longer builds the legacy `gvr` flavor that talks to the old PhoneVR +PC server, so it is not suitable to send upstream as it is. + +## 2. `alvr_client_core.h` + +The header used by this project was compared with the header generated by +`cargo xtask build-client-lib` (cbindgen) from a clean ALVR checkout at `dc84d683`. There were no +differences. The Gradle `buildClientLib` task regenerates this file on every build. ## 3. `alvr_main.cpp` API changes (v20 -> v21) | Area | v20 | v21 | |---|---|---| -| View params | Baked into `alvr_send_tracking`'s 2nd arg, recomputed every tick | Sent once via `alvr_send_view_params()` (head-local pose only), whenever eye offsets/FOV change | -| Head prediction offset | `alvr_get_head_prediction_offset_ns()` | **Removed from the C API.** Now a fixed heuristic (`ESTIMATED_PREDICTION_OFFSET_NS`, 30ms) - see the `TODO(upstream)` comment. Consider upstreaming a C export for `ClientCoreContext::get_total_prediction_offset()`. | -| `alvr_resume_opengl` | 5 args (trailing `always_direct` bool) | 4 args, bool dropped | -| `alvr_render_lobby_opengl` | `AlvrViewInput[2]` | `AlvrLobbyViewParams[2]` (adds `swapchain_index` wrapper) + `render_background: bool` | -| `alvr_render_stream_opengl` | `(buffer, swapchain_indices[2])` | `(buffer, AlvrStreamViewParams[2], foveation_center_shifts)` - now requires per-eye `reprojection_rotation` + `fov`, computed via the new `alvr_rotation_delta()` helper. This is genuinely new capability: real head-rotation reprojection, which the Cardboard distortion renderer never had on its own. | -| `alvr_get_frame` | Returned `(view_params, buffer)` + timestamp | Returns `(timestamp, buffer)` only. View params/foveation centers for that frame are now fetched separately via `alvr_report_compositor_start(timestamp, &metadata)`, which can return `false` (no match) - the caller then reuses the last known-good metadata (implemented via `CTX.lastFrameMetadata`/`hasFrameMetadata`). | -| Foveation params | Parsed by hand from `alvr_get_settings_json()` | Delivered directly on `AlvrEvent::StreamingStarted` (`enable_foveated_encoding`, `foveated_encoding`) | -| `AlvrStreamConfig` | No upscaling fields | New `enable_upscaling`/`upscaling_edge_*`/`upscale_factor` fields (left disabled here) | -| `AlvrClientCapabilities` | - | New `max_view_width/height`, `prefer_10bit`, `preferred_encoding_gamma`, `prefer_hdr` (left at conservative defaults here) | -| Enum style | Flat C enums (`ALVR_EVENT_STREAMING_STARTED`, `ALVR_LOG_LEVEL_ERROR`) | **Unchanged** - still flat/unscoped despite `AlvrEvent` now being a genuine tagged union in Rust. Confirmed against the untouched `utils.h`, which already uses this convention. | - -Validated with a syntax-only compile (`g++ -fsyntax-only -Wall -Wextra`) against hand-written -stub headers for `jni.h`, `android/log.h`, `GLES3/gl3.h`, and `cardboard.h`, since the real -Android NDK/Cardboard SDK aren't available in this environment either. This caught two real bugs -before they'd have hit a real build: enums accidentally declared as scoped `enum class` instead of -plain C enums, and reversed pointer constness on `swapchain_textures`. It cannot substitute for -building against the real NDK + regenerated header, since the stubs are approximations. - -## Not done yet (separate, later step) - -ARCore integration and the rotational-drift complementary filter. `getPose()` in `alvr_main.cpp` -is explicitly commented as the extension point for that work - it currently returns Cardboard's -gyro-only orientation with position pinned to the origin (3DoF). +| View params | Part of `alvr_send_tracking`'s arguments, recomputed every tick | Sent separately via `alvr_send_view_params()` (head-local pose), when eye offsets or FOV change | +| Head prediction offset | `alvr_get_head_prediction_offset_ns()` | Removed from the C API. A fixed estimate (`ESTIMATED_PREDICTION_OFFSET_NS`) is used instead. Consider upstreaming a C export for `ClientCoreContext::get_total_prediction_offset()`. | +| `alvr_resume_opengl` | 5 arguments (trailing `always_direct` bool) | 4 arguments | +| `alvr_render_lobby_opengl` | `AlvrViewInput[2]` | `AlvrLobbyViewParams[2]` (adds `swapchain_index`) plus a `render_background` bool | +| `alvr_render_stream_opengl` | `(buffer, swapchain_indices[2])` | `(buffer, AlvrStreamViewParams[2], foveation_center_shifts)`. Needs a per-eye `reprojection_rotation` and `fov`, computed with the new `alvr_rotation_delta()` helper. | +| `alvr_get_frame` | Returned view params, buffer and timestamp | Returns timestamp and buffer only. View params and foveation centers come from `alvr_report_compositor_start(timestamp, &metadata)`, which can return `false`; the caller then reuses the last good metadata. | +| Foveation parameters | Parsed by hand from `alvr_get_settings_json()` | Delivered on the `StreamingStarted` event | +| `AlvrStreamConfig` | No upscaling fields | New `enable_upscaling` / `upscaling_*` fields (left disabled) | +| Decoder | Created implicitly | Must be created by the client when `ALVR_EVENT_DECODER_CONFIG` arrives (see section 4) | + +## 4. Problems found while getting video working + +### 4.1 Host textures were invisible to the Rust GL context + +- **Symptom:** the lobby never rendered correctly. Rust logged `texture 2 is NULL` and + `framebuffer is not complete`. When streaming started, the app aborted (`SIGABRT`) inside + `StreamRenderer::new`, in `check_error`, on a leftover GL error `0x506`. +- **Cause:** `alvr_initialize_opengl()` calls `GraphicsContext::new_gl()`, which builds its own + wgpu GL instance. Its EGL context was created without a share context, so the swapchain + textures that PhoneVR creates in its own context did not exist there. +- **Fix:** a small patch to `wgpu-hal` 25.0.2 (`src/gles/egl.rs`) passes the thread's current EGL + context as the share context to the three `create_context` calls. It relies on PhoneVR calling + `alvr_initialize_opengl()` while its own context is current (it does, in `surfaceCreatedNative`). + The patch is in `code/mobile/android/PhoneVR/patches/wgpu-hal-25.0.2-share-egl-context.patch`. + It also contains some `ALVR GLES` debug log lines that can be dropped; the essential change is + replacing `None` with `share_context` in the `create_context` calls. +- **Related workaround:** the `alvr_*_opengl()` calls make Rust's context current and leave the + thread with none, so `alvr_main.cpp` restores the app's EGL state afterwards (`EglStateGuard`). + +### 4.2 The decoder was never created + +- **Symptom:** the lobby froze after connecting, no video frames arrived, and the stream renderer + never drew a frame, while SteamVR still received poses. +- **Cause:** the client did not handle `ALVR_EVENT_DECODER_CONFIG`. +- **Fix:** on that event, read the config bytes with `alvr_get_decoder_config` (call once with + `nullptr` for the size, then again to fill a buffer), fill an `AlvrDecoderConfig` using the codec + from the event, and call `alvr_create_decoder`. Ignore later decoder-config events until + reconnection, as the header says. Call `alvr_destroy_decoder` on `STREAMING_STOPPED`. + +### 4.3 Hardware decoder rejected the first input without MediaCodec options + +- **Symptom:** the Qualcomm decoder (`c2.qti.avc.decoder`) logged `Unsupported input buffer` + on the first input, ALVR retried, then restarted the connection with `Too many decoder errors`. + The software decoder worked. +- **Fix:** pass the same options the stock client gets from the server defaults in + `AlvrDecoderConfig.options`: `operating-rate` = `INT32_MAX`, `priority` = 0 and + `vendor.qti-ext-dec-low-latency.enable` = 1 (all int32). With these, hardware decoding works for + H.264 and HEVC. The generic `low-latency` key is deliberately not set; ALVR's own settings code + notes it crashes some Android phones. + +## 5. Setting up the ALVR side + +1. Check out ALVR at the commit above and build as usual (`prepare-alvr-deps.sh`, which runs + `cargo xtask build-client-lib`). `JAVA_HOME`, `ANDROID_HOME` and `ANDROID_NDK_HOME` must be set, + and an Android SDK platform and build-tools must be installed. +2. Download the `wgpu-hal` 25.0.2 crate (`https://static.crates.io/crates/wgpu-hal/wgpu-hal-25.0.2.crate`), + unpack it, and apply the patch from inside the unpacked folder: + `patch -p1 < /wgpu-hal-25.0.2-share-egl-context.patch` +3. In ALVR's top-level `Cargo.toml`, add: + ```toml + [patch.crates-io] + wgpu-hal = { path = "" } + ``` + The path used during development was absolute and specific to one machine; use your own. + +## 6. Known issues and not done yet + +- **ARCore and drift correction:** not integrated on this branch. `getPose()` in `alvr_main.cpp` + is the extension point; it currently returns Cardboard's gyro-only orientation with position + fixed at the origin. +- **Stuttering:** present in v20 as well. A fix exists on the author's separate v20 ARCore branch + and has not been merged here. Weak Wi-Fi also contributed in testing. +- **Stale decoder-config event:** a `DECODER_CONFIG` event arrives after each `STREAMING_STOPPED`, + and the handler creates a decoder while no stream is active. Harmless so far, but it should be + tidied. +- **Patch path:** the `wgpu-hal` patch is referenced by an absolute path on the development + machine, which must be replaced before others can build this. +- **CI:** `.github/workflows/build.yml` and `release.yml` still reference the removed + `gvr` / `noGvr` Gradle tasks and will need simplifying before CI can pass. +- **Likely obsolete:** `driver/PVRServer` and `code/windows` (the PC side of the legacy + protocol). Left untouched. From ee433f614aa56630306d660a45bcf04e5c0a524f Mon Sep 17 00:00:00 2001 From: Rob Oates Date: Mon, 5 Oct 2026 01:44:17 +0900 Subject: [PATCH 04/29] Make gradlew executable --- code/mobile/android/PhoneVR/gradlew | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 code/mobile/android/PhoneVR/gradlew diff --git a/code/mobile/android/PhoneVR/gradlew b/code/mobile/android/PhoneVR/gradlew old mode 100644 new mode 100755 From 2622995711154ad4dd4839473da6bc080f8fd016 Mon Sep 17 00:00:00 2001 From: Rob Oates Date: Mon, 5 Oct 2026 01:47:56 +0900 Subject: [PATCH 05/29] Point ALVR submodule at 9f118394 (21.0.0-dev14) --- code/mobile/android/PhoneVR/ALVR | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/code/mobile/android/PhoneVR/ALVR b/code/mobile/android/PhoneVR/ALVR index 1d121fa7..9f118394 160000 --- a/code/mobile/android/PhoneVR/ALVR +++ b/code/mobile/android/PhoneVR/ALVR @@ -1 +1 @@ -Subproject commit 1d121fa7e0761ef26473c6b3f3fbd9b6051ce033 +Subproject commit 9f11839431f95f0df6764790d9a16f8e308a0d79 From a0e93d7cbad8325a45afdf4da4a108baedab9e2e Mon Sep 17 00:00:00 2001 From: Rob Oates Date: Mon, 5 Oct 2026 02:30:08 +0900 Subject: [PATCH 06/29] Fix minor typos --- code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp index fcaaf60a..bcb0061d 100644 --- a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp +++ b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp @@ -212,7 +212,7 @@ void sendViewParams() { void inputThread() { auto deadline = std::chrono::steady_clock::now(); -ff + info("inputThread: thread staring..."); while (CTX.streaming) { auto targetTimestampNs = GetBootTimeNano() + ESTIMATED_PREDICTION_OFFSET_NS; @@ -220,13 +220,13 @@ ff CTX.deviceMotion.device_id = HEAD_ID; CTX.deviceMotion.pose = getPose(targetTimestampNs); - alvr_send_tracking(targetTimestampNs, &CTX.deviceMotion, 1, nullptr, nullptr);. + alvr_send_tracking(targetTimestampNs, &CTX.deviceMotion, 1, nullptr, nullptr); deadline += std::chrono::nanoseconds((uint64_t) (1e9 / 60.f / 3)); std::this_thread::sleep_until(deadline); } } -.. + extern "C" JNIEXPORT jint JNI_OnLoad(JavaVM *vm, void *) { CTX.javaVm = vm; return JNI_VERSION_1_6; From 54bcaaefd2d5d1a6322d83dcd7b36a3457f10eee Mon Sep 17 00:00:00 2001 From: Rob Oates Date: Mon, 5 Oct 2026 20:30:46 +0900 Subject: [PATCH 07/29] ARCore build plumbing: link ARCore, pass enableARCore through JNI (unused yet) --- code/mobile/android/PhoneVR/.gitignore | 1 + code/mobile/android/PhoneVR/app/CMakeLists.txt | 10 +++++++++- .../main/java/viritualisres/phonevr/ALVRActivity.java | 4 ++-- code/mobile/android/PhoneVR/prepare-alvr-deps.sh | 4 ++++ 4 files changed, 16 insertions(+), 3 deletions(-) diff --git a/code/mobile/android/PhoneVR/.gitignore b/code/mobile/android/PhoneVR/.gitignore index fa0251e2..dd8cdb5d 100644 --- a/code/mobile/android/PhoneVR/.gitignore +++ b/code/mobile/android/PhoneVR/.gitignore @@ -48,3 +48,4 @@ google-services.json # Android Profiling *.hprof +arcore-android-sdk-main/ diff --git a/code/mobile/android/PhoneVR/app/CMakeLists.txt b/code/mobile/android/PhoneVR/app/CMakeLists.txt index f50969b5..f0db9551 100644 --- a/code/mobile/android/PhoneVR/app/CMakeLists.txt +++ b/code/mobile/android/PhoneVR/app/CMakeLists.txt @@ -11,6 +11,13 @@ add_library(native-lib-alvr SHARED src/main/cpp/alvr_main.cpp ) +# Import the ARCore (Google Play Services for AR) library. +add_library(arcore SHARED IMPORTED) +set_target_properties(arcore PROPERTIES IMPORTED_LOCATION + ${ARCORE_LIBPATH}/${ANDROID_ABI}/libarcore_sdk_c.so + INTERFACE_INCLUDE_DIRECTORIES ${ARCORE_INCLUDE} +) + set(libs_dir ${CMAKE_CURRENT_SOURCE_DIR}/libraries) set(alvr_build_dir ${CMAKE_CURRENT_SOURCE_DIR}/../ALVR/build/alvr_client_core) @@ -18,10 +25,11 @@ target_include_directories(native-lib-alvr # ALVR Headers PUBLIC ${alvr_build_dir} PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/../cardboard + PUBLIC ${ARCORE_INCLUDE} ) target_link_libraries(native-lib-alvr - log android EGL GLESv3 mediandk + log android EGL GLESv3 mediandk arcore ${alvr_build_dir}/${ANDROID_ABI}/libalvr_client_core.so ${libs_dir}/jni/${ANDROID_ABI}/libGfxPluginCardboard.so ) diff --git a/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/ALVRActivity.java b/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/ALVRActivity.java index b5b09f56..1b6cf0a4 100644 --- a/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/ALVRActivity.java +++ b/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/ALVRActivity.java @@ -105,7 +105,7 @@ public void onCreate(Bundle savedInstance) { float refreshRate = display.getRefreshRate(); Log.i(TAG, "Refresh rate: " + refreshRate); - initializeNative(width, height, refreshRate); + initializeNative(width, height, refreshRate, true); setContentView(R.layout.activity_vr); glView = findViewById(R.id.surface_view); @@ -285,7 +285,7 @@ private void setImmersiveSticky() { } private native void initializeNative( - int screenWidth, int screenHeight, float screenRefreshRate); + int screenWidth, int screenHeight, float screenRefreshRate, boolean enableARCore); private native void destroyNative(); diff --git a/code/mobile/android/PhoneVR/prepare-alvr-deps.sh b/code/mobile/android/PhoneVR/prepare-alvr-deps.sh index d1df3391..0cb2c563 100755 --- a/code/mobile/android/PhoneVR/prepare-alvr-deps.sh +++ b/code/mobile/android/PhoneVR/prepare-alvr-deps.sh @@ -39,3 +39,7 @@ mv "${CARB_REPO_NAME}/sdk/build/outputs/aar/sdk-release.aar" cardboard/cardboard cp "${CARB_REPO_NAME}/sdk/include/cardboard.h" cardboard/cardboard.h rm -r "${CARB_REPO_NAME}" +rm -rf "arcore-android-sdk-main" +curl -sLS https://github.com/google-ar/arcore-android-sdk/archive/refs/heads/main.zip > download.zip +unzip download.zip +rm download.zip From f083ec3bb4101ffb3c46907c6dbd4250f7ef3b0f Mon Sep 17 00:00:00 2001 From: Rob Oates Date: Mon, 5 Oct 2026 20:32:01 +0900 Subject: [PATCH 08/29] ARCore build plumbing: link ARCore, pass enableARCore through JNI (unused yet) --- code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp index bcb0061d..9b345ef6 100644 --- a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp +++ b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp @@ -233,8 +233,9 @@ extern "C" JNIEXPORT jint JNI_OnLoad(JavaVM *vm, void *) { } extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_initializeNative( - JNIEnv *env, jobject obj, jint screenWidth, jint screenHeight, jfloat refreshRate) { + JNIEnv *env, jobject obj, jint screenWidth, jint screenHeight, jfloat refreshRate,jboolean enableARCore) { CTX.javaContext = env->NewGlobalRef(obj); + info("initializeNative: enableARCore=%d (not used yet)", (int) enableARCore); uint32_t viewWidth = (std::max(screenWidth, screenHeight) / 2) & ~15u; // multiple of 16 uint32_t viewHeight = std::min(screenWidth, screenHeight) & ~7u; // multiple of 8 From e592b18dfae0fad84da0bc17145683174d8a38e9 Mon Sep 17 00:00:00 2001 From: Rob Oates Date: Tue, 6 Oct 2026 11:35:52 +0900 Subject: [PATCH 09/29] ARCore session, camera texture and per-frame diagnostics (step 2a) --- .../PhoneVR/app/src/main/cpp/alvr_main.cpp | 167 +++++++++++++++++- 1 file changed, 166 insertions(+), 1 deletion(-) diff --git a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp index 9b345ef6..3e4dda28 100644 --- a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp +++ b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp @@ -1,4 +1,5 @@ #include "alvr_client_core.h" +#include "arcore_c_api.h" #include "cardboard.h" #include #include @@ -8,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -93,6 +95,15 @@ struct NativeContext { CardboardLensDistortion *lensDistortion = nullptr; CardboardDistortionRenderer *distortionRenderer = nullptr; + // ARCore (positional tracking). ARCore calls that touch the session must not run + // concurrently, so every use of arSession is guarded by arMutex. ArSession_update() must + // additionally run on the render thread, with the app's EGL context current. + bool arcoreEnabled = false; + ArSession *arSession = nullptr; + ArFrame *arFrame = nullptr; + GLuint arTexture = 0; + std::mutex arMutex; + int screenWidth = 0; int screenHeight = 0; @@ -192,6 +203,63 @@ AlvrPose getPose(uint64_t timestampNs) { return pose; } +// STEP 2a (diagnostic only): update ARCore once per render frame and log what it reports about +// once per second. The result is NOT used for tracking yet; this only checks that the session +// runs on the render thread with the app's EGL context current. Must be called from the render +// thread. +static void pollARCoreDiagnostics() { + if (!CTX.arcoreEnabled || CTX.arTexture == 0) { + return; + } + if (eglGetCurrentContext() == EGL_NO_CONTEXT) { + info("ARCore ERROR: pollARCoreDiagnostics called without an EGL context"); + return; + } + + std::lock_guard lock(CTX.arMutex); + if (CTX.arSession == nullptr || CTX.arFrame == nullptr) { + return; + } + + static uint64_t lastLogNs = 0; + uint64_t nowNs = GetBootTimeNano(); + bool shouldLog = nowNs - lastLogNs >= 1000000000ULL; + + ArStatus status = ArSession_update(CTX.arSession, CTX.arFrame); + if (status != AR_SUCCESS) { + if (shouldLog) { + lastLogNs = nowNs; + info("ARCore ERROR: ArSession_update failed (status %d)", (int) status); + } + return; + } + + ArCamera *arCamera = nullptr; + ArFrame_acquireCamera(CTX.arSession, CTX.arFrame, &arCamera); + + ArTrackingState trackingState; + ArCamera_getTrackingState(CTX.arSession, arCamera, &trackingState); + + if (shouldLog) { + lastLogNs = nowNs; + if (trackingState == AR_TRACKING_STATE_TRACKING) { + ArPose *arPose = nullptr; + ArPose_create(CTX.arSession, nullptr, &arPose); + ArCamera_getDisplayOrientedPose(CTX.arSession, arCamera, arPose); + // {qx, qy, qz, qw, tx, ty, tz} + float raw[7] = {}; + ArPose_getPoseRaw(CTX.arSession, arPose, raw); + ArPose_destroy(arPose); + info("ARCore: tracking, position=(%.3f, %.3f, %.3f) quat=(%.3f, %.3f, %.3f, %.3f)", + raw[4], raw[5], raw[6], raw[0], raw[1], raw[2], raw[3]); + } else { + info("ARCore: not tracking (state %d)", (int) trackingState); + } + } + + ArCamera_release(arCamera); +} + // Local-space (head-at-origin) per-eye view parameters: just the eye offset and FOV. These rarely // change (only when the lens distortion / viewer profile is recalculated), so this is sent once // via alvr_send_view_params() rather than on every tracking update - unlike the v20 API, the @@ -235,7 +303,7 @@ extern "C" JNIEXPORT jint JNI_OnLoad(JavaVM *vm, void *) { extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_initializeNative( JNIEnv *env, jobject obj, jint screenWidth, jint screenHeight, jfloat refreshRate,jboolean enableARCore) { CTX.javaContext = env->NewGlobalRef(obj); - info("initializeNative: enableARCore=%d (not used yet)", (int) enableARCore); + info("initializeNative: enableARCore=%d", (int) enableARCore); uint32_t viewWidth = (std::max(screenWidth, screenHeight) / 2) & ~15u; // multiple of 16 uint32_t viewHeight = std::min(screenWidth, screenHeight) & ~7u; // multiple of 8 @@ -267,6 +335,46 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_initia Cardboard_initializeAndroid(CTX.javaVm, CTX.javaContext); CTX.headTracker = CardboardHeadTracker_create(); + + CTX.arcoreEnabled = (bool) enableARCore; + if (CTX.arcoreEnabled) { + if (ArSession_create(env, CTX.javaContext, &CTX.arSession) != AR_SUCCESS) { + info("ARCore ERROR: could not create session (is Google Play Services for AR " + "installed?); continuing without ARCore"); + CTX.arSession = nullptr; + CTX.arcoreEnabled = false; + } + } + + if (CTX.arcoreEnabled) { + ArConfig *arConfig = nullptr; + ArConfig_create(CTX.arSession, &arConfig); + + // Explicitly disable everything we don't need, to preserve CPU and battery. + ArConfig_setDepthMode(CTX.arSession, arConfig, AR_DEPTH_MODE_DISABLED); + ArConfig_setLightEstimationMode( + CTX.arSession, arConfig, AR_LIGHT_ESTIMATION_MODE_DISABLED); + ArConfig_setPlaneFindingMode( + CTX.arSession, arConfig, AR_PLANE_FINDING_MODE_HORIZONTAL_AND_VERTICAL); + ArConfig_setCloudAnchorMode(CTX.arSession, arConfig, AR_CLOUD_ANCHOR_MODE_DISABLED); + // Don't block waiting for a new camera image. + ArConfig_setUpdateMode(CTX.arSession, arConfig, AR_UPDATE_MODE_LATEST_CAMERA_IMAGE); + + ArStatus configureStatus = ArSession_configure(CTX.arSession, arConfig); + ArConfig_destroy(arConfig); + + if (configureStatus != AR_SUCCESS) { + info("ARCore ERROR: could not configure session (status %d); continuing without " + "ARCore", + (int) configureStatus); + ArSession_destroy(CTX.arSession); + CTX.arSession = nullptr; + CTX.arcoreEnabled = false; + } else { + ArFrame_create(CTX.arSession, &CTX.arFrame); + info("ARCore session created and configured"); + } + } } extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_destroyNative(JNIEnv *, @@ -280,12 +388,35 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_destro CTX.lensDistortion = nullptr; CardboardDistortionRenderer_destroy(CTX.distortionRenderer); CTX.distortionRenderer = nullptr; + + { + std::lock_guard lock(CTX.arMutex); + if (CTX.arFrame != nullptr) { + ArFrame_destroy(CTX.arFrame); + CTX.arFrame = nullptr; + } + if (CTX.arSession != nullptr) { + ArSession_destroy(CTX.arSession); + CTX.arSession = nullptr; + } + CTX.arcoreEnabled = false; + } } extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_resumeNative(JNIEnv *, jobject) { CardboardHeadTracker_resume(CTX.headTracker); + { + std::lock_guard lock(CTX.arMutex); + if (CTX.arcoreEnabled && CTX.arSession != nullptr) { + ArStatus arStatus = ArSession_resume(CTX.arSession); + if (arStatus != AR_SUCCESS) { + info("ARCore ERROR: failed to resume session (status %d)", (int) arStatus); + } + } + } + CTX.renderingParamsChanged = true; uint8_t *buffer; @@ -313,6 +444,14 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_pauseN } CardboardHeadTracker_pause(CTX.headTracker); + + // Releases the camera. (The v20 code never paused the ARCore session.) + { + std::lock_guard lock(CTX.arMutex); + if (CTX.arcoreEnabled && CTX.arSession != nullptr) { + ArSession_pause(CTX.arSession); + } + } } extern "C" JNIEXPORT void JNICALL @@ -459,6 +598,29 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render } EGLCHK("after alvr_resume_opengl (restored)"); + // ARCore needs a GL texture to write camera frames into. We never draw the camera + // image, but ARCore requires one. If the GL context was recreated the old texture is + // already gone; otherwise delete it so it doesn't leak. + if (CTX.arcoreEnabled && CTX.arSession != nullptr) { + if (CTX.arTexture != 0 && !CTX.glContextRecreated) { + glDeleteTextures(1, &CTX.arTexture); + } + GLuint arTextureIds[1] = {0}; + glGenTextures(1, arTextureIds); + GL(glBindTexture(GL_TEXTURE_2D, arTextureIds[0])); + GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE)); + GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE)); + GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR)); + GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR)); + + { + std::lock_guard lock(CTX.arMutex); + CTX.arTexture = arTextureIds[0]; + ArSession_setCameraTextureName(CTX.arSession, CTX.arTexture); + } + info("ARCore camera texture initialized: %u", (unsigned) CTX.arTexture); + } + CTX.renderingParamsChanged = false; CTX.glContextRecreated = false; } @@ -633,6 +795,9 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render // before `CardboardEyeTextureDescription viewsDescs[2] = {};` EGLCHK("after event loop"); + // Step 2a: diagnostics only; the ARCore result is not used for tracking yet. + pollARCoreDiagnostics(); + CardboardEyeTextureDescription viewsDescs[2] = {}; for (auto &viewsDesc : viewsDescs) { viewsDesc.left_u = 0.0; From a61616512849ac3987ed0ef28734aa1a61484aad Mon Sep 17 00:00:00 2001 From: Rob Oates Date: Tue, 6 Oct 2026 14:40:09 +0900 Subject: [PATCH 10/29] ARCore position and orientation correction (step 2b) --- .../PhoneVR/app/src/main/cpp/alvr_main.cpp | 208 +++++++++++++++--- 1 file changed, 177 insertions(+), 31 deletions(-) diff --git a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp index 3e4dda28..b5d2c4eb 100644 --- a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp +++ b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include @@ -87,6 +88,21 @@ const uint64_t VSYNC_QUEUE_INTERVAL_NS = 50e6; const float FLOOR_HEIGHT = 1.5; const int MAXIMUM_TRACKING_FRAMES = 360; +// ---- ARCore tuning (values carried over from the v20 branch; tuned on one phone only) ---- +// Use ARCore's position as the head position (otherwise position stays at the origin: 3DoF). +constexpr bool ARCORE_USE_POSITION = true; +// Slowly pull Cardboard's drifting orientation toward ARCore's (complementary filter). +constexpr bool ARCORE_USE_ORIENTATION_CORRECTION = true; +// How fast the correction follows ARCore. Larger = slower and smoother. +constexpr float ARCORE_CORRECTION_TIME_CONSTANT_S = 3.0f; +// If Cardboard and ARCore disagree by more than this, ignore the sample as a bad measurement. +constexpr float ARCORE_MAX_ORIENTATION_ERROR_DEG = 15.0f; +// Ignore the update after a gap longer than this (pause, lifecycle change, debugger). +constexpr uint64_t ARCORE_MAX_CORRECTION_GAP_NS = 500000000ULL; +// Fixed rotation between ARCore's display-oriented camera frame and Cardboard's frame +// (landscape-left). Found by calibration on one phone; other phones may need other values. +const AlvrQuat ARCORE_TO_CARDBOARD = {-0.0057f, -0.0031f, 0.7076f, 0.7066f}; + struct NativeContext { JavaVM *javaVm = nullptr; jobject javaContext = nullptr; @@ -104,6 +120,14 @@ struct NativeContext { GLuint arTexture = 0; std::mutex arMutex; + // Pose state written by the render thread (updateARCoreTracking) and read by any thread + // that calls getPose(), for example the input thread. Guarded by poseMutex. + std::mutex poseMutex; + bool arPositionValid = false; + float arPosition[3] = {0.f, 0.f, 0.f}; + AlvrQuat orientationCorrection = {0.f, 0.f, 0.f, 1.f}; + uint64_t lastCorrectionNs = 0; + int screenWidth = 0; int screenHeight = 0; @@ -183,36 +207,107 @@ AlvrFov getFov(CardboardEye eye) { return fov; } -// Head pose, in world/tracking space (not head-relative). Rotation currently comes straight from -// the Cardboard SDK's gyro-only sensor fusion, so it has no absolute position and will drift over -// time - this is the exact function that the planned ARCore + complementary filter work (a -// separate, later step) will extend to fuse in ARCore's 6DoF pose. -AlvrPose getPose(uint64_t timestampNs) { - AlvrPose pose = {}; +// Quaternion helpers for the orientation correction (same maths as the v20 branch). +static AlvrQuat quatMultiply(const AlvrQuat &a, const AlvrQuat &b) { + return {a.w * b.x + a.x * b.w + a.y * b.z - a.z * b.y, + a.w * b.y - a.x * b.z + a.y * b.w + a.z * b.x, + a.w * b.z + a.x * b.y - a.y * b.x + a.z * b.w, + a.w * b.w - a.x * b.x - a.y * b.y - a.z * b.z}; +} + +// Inverse of a unit quaternion. +static AlvrQuat quatInverse(const AlvrQuat &q) { return {-q.x, -q.y, -q.z, q.w}; } + +static AlvrQuat normalizeQuat(AlvrQuat q) { + float len = std::sqrt(q.x * q.x + q.y * q.y + q.z * q.z + q.w * q.w); + if (len < 1e-6f) { + return {0.f, 0.f, 0.f, 1.f}; + } + return {q.x / len, q.y / len, q.z / len, q.w / len}; +} + +// q and -q are the same rotation, so use abs(w). +static float quatAngleDegrees(const AlvrQuat &q) { + float w = std::clamp(std::abs(q.w), 0.0f, 1.0f); + return 2.0f * std::acos(w) * 180.0f / (float) M_PI; +} +static AlvrQuat slerpQuat(AlvrQuat a, AlvrQuat b, float t) { + a = normalizeQuat(a); + b = normalizeQuat(b); + + float dot = a.x * b.x + a.y * b.y + a.z * b.z + a.w * b.w; + + // Take the shortest path. + if (dot < 0.0f) { + b = {-b.x, -b.y, -b.z, -b.w}; + dot = -dot; + } + + // Very close: normalized linear interpolation is enough. + if (dot > 0.9995f) { + return normalizeQuat({a.x + t * (b.x - a.x), + a.y + t * (b.y - a.y), + a.z + t * (b.z - a.z), + a.w + t * (b.w - a.w)}); + } + + float theta = std::acos(dot); + float sinTheta = std::sin(theta); + float wa = std::sin((1.0f - t) * theta) / sinTheta; + float wb = std::sin(t * theta) / sinTheta; + + return normalizeQuat({wa * a.x + wb * b.x, + wa * a.y + wb * b.y, + wa * a.z + wb * b.z, + wa * a.w + wb * b.w}); +} + +// Cardboard's orientation at the given time, in the same convention the rest of this file uses. +static AlvrQuat cardboardOrientationAt(uint64_t timestampNs) { float pos[3]; float q[4]; CardboardHeadTracker_getPose(CTX.headTracker, (int64_t) timestampNs, kLandscapeLeft, pos, q); + return normalizeQuat(inverseQuat(AlvrQuat{q[0], q[1], q[2], q[3]})); +} - auto inverseOrientation = AlvrQuat{q[0], q[1], q[2], q[3]}; - pose.orientation = inverseQuat(inverseOrientation); +// Head pose, in world/tracking space (not head-relative). Safe to call from any thread. +// +// Orientation: Cardboard's fast gyro orientation, multiplied by a slowly changing correction that +// the render thread computes from ARCore (updateARCoreTracking). This is what removes the drift. +// Position: ARCore's last known position, or the origin until ARCore has tracked (3DoF). +AlvrPose getPose(uint64_t timestampNs) { + AlvrPose pose = {}; + + AlvrQuat orientation = cardboardOrientationAt(timestampNs); - // Cardboard-only tracking has no position signal (3DoF), so position stays at the origin - // for now. + { + std::lock_guard lock(CTX.poseMutex); + + if (ARCORE_USE_ORIENTATION_CORRECTION) { + orientation = normalizeQuat(quatMultiply(orientation, CTX.orientationCorrection)); + } + if (ARCORE_USE_POSITION && CTX.arPositionValid) { + for (int i = 0; i < 3; i++) { + pose.position[i] = CTX.arPosition[i]; + } + } + } + + pose.orientation = orientation; return pose; } -// STEP 2a (diagnostic only): update ARCore once per render frame and log what it reports about -// once per second. The result is NOT used for tracking yet; this only checks that the session -// runs on the render thread with the app's EGL context current. Must be called from the render -// thread. -static void pollARCoreDiagnostics() { +// Updates ARCore once per render frame and publishes what getPose() needs: the ARCore position +// and the orientation correction. Must be called from the render thread, with the app's EGL +// context current (ArSession_update requires it). +static void updateARCoreTracking() { if (!CTX.arcoreEnabled || CTX.arTexture == 0) { return; } if (eglGetCurrentContext() == EGL_NO_CONTEXT) { - info("ARCore ERROR: pollARCoreDiagnostics called without an EGL context"); + info("ARCore ERROR: updateARCoreTracking called without an EGL context"); return; } @@ -240,24 +335,75 @@ static void pollARCoreDiagnostics() { ArTrackingState trackingState; ArCamera_getTrackingState(CTX.arSession, arCamera, &trackingState); - if (shouldLog) { - lastLogNs = nowNs; - if (trackingState == AR_TRACKING_STATE_TRACKING) { - ArPose *arPose = nullptr; - ArPose_create(CTX.arSession, nullptr, &arPose); - ArCamera_getDisplayOrientedPose(CTX.arSession, arCamera, arPose); - // {qx, qy, qz, qw, tx, ty, tz} - float raw[7] = {}; - ArPose_getPoseRaw(CTX.arSession, arPose, raw); - ArPose_destroy(arPose); - info("ARCore: tracking, position=(%.3f, %.3f, %.3f) quat=(%.3f, %.3f, %.3f, %.3f)", - raw[4], raw[5], raw[6], raw[0], raw[1], raw[2], raw[3]); - } else { + if (trackingState != AR_TRACKING_STATE_TRACKING) { + // Not tracking: keep the last known position and correction. + if (shouldLog) { + lastLogNs = nowNs; info("ARCore: not tracking (state %d)", (int) trackingState); } + ArCamera_release(arCamera); + return; } + ArPose *arPose = nullptr; + ArPose_create(CTX.arSession, nullptr, &arPose); + ArCamera_getDisplayOrientedPose(CTX.arSession, arCamera, arPose); + // {qx, qy, qz, qw, tx, ty, tz} + float raw[7] = {}; + ArPose_getPoseRaw(CTX.arSession, arPose, raw); + ArPose_destroy(arPose); ArCamera_release(arCamera); + + // How far apart are Cardboard and ARCore right now? + AlvrQuat cardboardOrientation = + cardboardOrientationAt(nowNs + ESTIMATED_PREDICTION_OFFSET_NS); + AlvrQuat arcoreOrientation = normalizeQuat(AlvrQuat{raw[0], raw[1], raw[2], raw[3]}); + AlvrQuat arcoreInCardboard = + normalizeQuat(quatMultiply(arcoreOrientation, quatInverse(ARCORE_TO_CARDBOARD))); + // The rotation that would take Cardboard's orientation onto ARCore's. + AlvrQuat relative = + normalizeQuat(quatMultiply(quatInverse(cardboardOrientation), arcoreInCardboard)); + float errorDeg = quatAngleDegrees(relative); + + bool accepted = errorDeg < ARCORE_MAX_ORIENTATION_ERROR_DEG; + float correctionDeg = 0.0f; + + { + std::lock_guard poseLock(CTX.poseMutex); + + CTX.arPosition[0] = raw[4]; + CTX.arPosition[1] = raw[5]; + CTX.arPosition[2] = raw[6]; + CTX.arPositionValid = true; + + if (accepted) { + if (CTX.lastCorrectionNs == 0) { + CTX.lastCorrectionNs = nowNs; + } + uint64_t deltaNs = nowNs - CTX.lastCorrectionNs; + // Protect against an unexpectedly large gap (pause, debugger, lifecycle change). + if (deltaNs > 0 && deltaNs < ARCORE_MAX_CORRECTION_GAP_NS) { + float dt = (float) deltaNs * 1e-9f; + // Time-based gain, so the result doesn't depend on the frame rate. + float gain = 1.0f - std::exp(-dt / ARCORE_CORRECTION_TIME_CONSTANT_S); + CTX.orientationCorrection = + slerpQuat(CTX.orientationCorrection, relative, gain); + } + CTX.lastCorrectionNs = nowNs; + } else { + // Don't let a transient mismatch pull the correction toward a bad measurement. + CTX.lastCorrectionNs = nowNs; + } + + correctionDeg = quatAngleDegrees(CTX.orientationCorrection); + } + + if (shouldLog) { + lastLogNs = nowNs; + info("ARCore: tracking, position=(%.3f, %.3f, %.3f) | orientation error=%.2f deg, " + "correction=%.2f deg, sample %s", + raw[4], raw[5], raw[6], errorDeg, correctionDeg, accepted ? "accepted" : "rejected"); + } } // Local-space (head-at-origin) per-eye view parameters: just the eye offset and FOV. These rarely @@ -795,8 +941,8 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render // before `CardboardEyeTextureDescription viewsDescs[2] = {};` EGLCHK("after event loop"); - // Step 2a: diagnostics only; the ARCore result is not used for tracking yet. - pollARCoreDiagnostics(); + // Update ARCore on the render thread; getPose() then uses its position and correction. + updateARCoreTracking(); CardboardEyeTextureDescription viewsDescs[2] = {}; for (auto &viewsDesc : viewsDescs) { From ae2ce64c104c6a54512ccb79013a03051540574c Mon Sep 17 00:00:00 2001 From: Rob Oates Date: Tue, 6 Oct 2026 15:03:56 +0900 Subject: [PATCH 11/29] Reuse the last decoded frame when no new frame arrives (fixes stutter) --- .../PhoneVR/app/src/main/cpp/alvr_main.cpp | 57 ++++++++++++++----- 1 file changed, 44 insertions(+), 13 deletions(-) diff --git a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp index b5d2c4eb..7db395fe 100644 --- a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp +++ b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp @@ -152,6 +152,13 @@ struct NativeContext { bool hasFrameMetadata = false; AlvrVideoFrameMetadata lastFrameMetadata = {}; + // The last decoded video frame, kept so it can be shown again (re-oriented to the current + // head pose) on display refreshes where no new video frame has arrived. The buffer belongs + // to the decoder, so this is cleared whenever the decoder is created or destroyed or a + // stream starts or stops. + void *lastStreamBuffer = nullptr; + uint64_t lastStreamTimestampNs = 0; + NativeContext() { memset(&fovArr, 0, sizeof(fovArr)); memset(&deviceMotion, 0, sizeof(deviceMotion)); @@ -802,6 +809,7 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render // manual settings-JSON parsing needed here (settings.video.foveated_encoding, as // the v20 code did). CTX.hasFrameMetadata = false; + CTX.lastStreamBuffer = nullptr; GL(glGenTextures(2, CTX.streamTextures)); @@ -917,6 +925,8 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render "software=%d", (int) decoderConfig.codec, (unsigned long long) configSize, (int) FORCE_SOFTWARE_DECODER); + // The saved frame belongs to the previous decoder, if there was one. + CTX.lastStreamBuffer = nullptr; alvr_create_decoder(decoderConfig); g_decoderCreated = true; } @@ -927,6 +937,8 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render CTX.streaming = false; CTX.inputThread.join(); + // The saved frame belongs to the decoder that is about to be destroyed. + CTX.lastStreamBuffer = nullptr; if (g_decoderCreated) { alvr_destroy_decoder(); g_decoderCreated = false; @@ -956,22 +968,38 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render void *streamHardwareBuffer = nullptr; uint64_t timestampNs = 0; - if (!alvr_get_frame(×tampNs, &streamHardwareBuffer)) { - return; - } + // A new decoded frame is not available on every display refresh: the video usually + // arrives more slowly than the display runs. Returning here without drawing leaves the + // screen showing stale or blank content, which looks like stuttering. Instead keep + // showing the last frame, re-oriented to the current head pose by the reprojection + // below. + bool isNewFrame = alvr_get_frame(×tampNs, &streamHardwareBuffer); - // v21 no longer returns view params from alvr_get_frame() directly; fetch the - // server-aligned metadata (view params + foveation centers) matched to this frame here - // instead. If there's no match for this exact frame, fall back to the last known-good - // metadata, per the API doc for this function. AlvrVideoFrameMetadata metadata; - if (alvr_report_compositor_start(timestampNs, &metadata)) { - CTX.lastFrameMetadata = metadata; - CTX.hasFrameMetadata = true; - } else if (CTX.hasFrameMetadata) { + if (isNewFrame) { + CTX.lastStreamBuffer = streamHardwareBuffer; + CTX.lastStreamTimestampNs = timestampNs; + + // v21 no longer returns view params from alvr_get_frame() directly; fetch the + // server-aligned metadata (view params + foveation centers) matched to this frame + // here instead. If there's no match for this exact frame, fall back to the last + // known-good metadata, per the API doc for this function. + if (alvr_report_compositor_start(timestampNs, &metadata)) { + CTX.lastFrameMetadata = metadata; + CTX.hasFrameMetadata = true; + } else if (CTX.hasFrameMetadata) { + metadata = CTX.lastFrameMetadata; + } else { + // No metadata at all yet (first frame) - nothing sensible to reproject against. + return; + } + } else if (CTX.lastStreamBuffer != nullptr && CTX.hasFrameMetadata) { + // No new frame: show the previous one again, with the metadata that belongs to it. + streamHardwareBuffer = CTX.lastStreamBuffer; + timestampNs = CTX.lastStreamTimestampNs; metadata = CTX.lastFrameMetadata; } else { - // No metadata at all yet (first frame) - nothing sensible to reproject against. + // Nothing has been decoded yet. return; } @@ -998,7 +1026,10 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render alvr_render_stream_opengl(streamHardwareBuffer, streamViewParams, foveationCenters)); EGLCHK("after alvr_render_stream_opengl()"); - alvr_report_submit(timestampNs, 0); + // The statistics are per decoded frame, so only report frames that are new. + if (isNewFrame) { + alvr_report_submit(timestampNs, 0); + } viewsDescs[0].texture = CTX.streamTextures[0]; viewsDescs[1].texture = CTX.streamTextures[1]; From b9276af91feb99e4ac6cff1610a58157de8165a1 Mon Sep 17 00:00:00 2001 From: Rob Oates Date: Tue, 6 Oct 2026 17:53:23 +0900 Subject: [PATCH 12/29] Fix floor height --- .../PhoneVR/app/src/main/cpp/alvr_main.cpp | 160 +++++++++++++++++- 1 file changed, 156 insertions(+), 4 deletions(-) diff --git a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp index 7db395fe..8678e84f 100644 --- a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp +++ b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp @@ -103,6 +103,24 @@ constexpr uint64_t ARCORE_MAX_CORRECTION_GAP_NS = 500000000ULL; // (landscape-left). Found by calibration on one phone; other phones may need other values. const AlvrQuat ARCORE_TO_CARDBOARD = {-0.0057f, -0.0031f, 0.7076f, 0.7066f}; +// ---- ARCore head height ---- +// How the vertical position is derived: +// 0 = relative to where ARCore started (height starts at 0, as in the earlier builds). +// 1 = height above the detected floor (the v20 behaviour). Until a floor is found it is the +// same as 0, so the height jumps the moment the floor is first detected. +// 2 = like 1, but until a floor is found it assumes the headset started +// ARCORE_ASSUMED_START_HEIGHT_M above the floor, and it eases toward the measured floor +// height instead of jumping. +constexpr int ARCORE_HEIGHT_MODE = 2; +// Mode 2 only: assumed height above the floor at the start, before any floor has been detected. +constexpr float ARCORE_ASSUMED_START_HEIGHT_M = 1.5f; +// Mode 2 only: how slowly the height eases toward the measured floor. Larger = slower. +constexpr float ARCORE_FLOOR_BLEND_TIME_CONSTANT_S = 2.0f; +// Added to the height in every mode. Use it to tune the height by hand (negative = lower). +constexpr float ARCORE_HEIGHT_OFFSET_M = 0.0f; +// A newly detected plane has to be this much lower than the current floor to replace it. +constexpr float ARCORE_FLOOR_REANCHOR_MARGIN_M = 0.03f; + struct NativeContext { JavaVM *javaVm = nullptr; jobject javaContext = nullptr; @@ -117,6 +135,12 @@ struct NativeContext { bool arcoreEnabled = false; ArSession *arSession = nullptr; ArFrame *arFrame = nullptr; + // Anchor on the lowest horizontal plane found so far (taken to be the floor). An anchor keeps + // following the floor when ARCore adjusts its world frame. Render thread only (arMutex). + ArAnchor *arFloorAnchor = nullptr; + // Mode 2: how far above the floor ARCore's origin is, eased toward the measured value. + float floorOffsetM = ARCORE_ASSUMED_START_HEIGHT_M; + uint64_t lastFloorBlendNs = 0; GLuint arTexture = 0; std::mutex arMutex; @@ -306,6 +330,101 @@ AlvrPose getPose(uint64_t timestampNs) { return pose; } +// Height (y) of an anchor in ARCore's world frame. +static float anchorHeight(const ArAnchor *anchor) { + ArPose *pose = nullptr; + ArPose_create(CTX.arSession, nullptr, &pose); + ArAnchor_getPose(CTX.arSession, anchor, pose); + float raw[7] = {}; + ArPose_getPoseRaw(CTX.arSession, pose, raw); + ArPose_destroy(pose); + return raw[5]; +} + +// Looks for the floor: the lowest horizontal, upward-facing plane ARCore has found so far. When +// one is known, returns true and sets *outFloorY to the floor's height in ARCore's world frame +// (read from an anchor, so it follows any adjustment ARCore makes to that frame). +// Render thread only, with arMutex held and a successful ArSession_update behind it. +static bool updateFloorAnchor(float *outFloorY) { + ArTrackableList *planes = nullptr; + ArTrackableList_create(CTX.arSession, &planes); + ArFrame_getUpdatedTrackables(CTX.arSession, CTX.arFrame, AR_TRACKABLE_PLANE, planes); + int32_t planeCount = 0; + ArTrackableList_getSize(CTX.arSession, planes, &planeCount); + + // Is the current floor anchor still usable? + float floorY = 0.0f; + bool haveFloor = false; + if (CTX.arFloorAnchor != nullptr) { + ArTrackingState anchorState; + ArAnchor_getTrackingState(CTX.arSession, CTX.arFloorAnchor, &anchorState); + if (anchorState == AR_TRACKING_STATE_STOPPED) { + // ARCore has given up on this anchor, so forget it. + ArAnchor_release(CTX.arFloorAnchor); + CTX.arFloorAnchor = nullptr; + } else if (anchorState == AR_TRACKING_STATE_TRACKING) { + floorY = anchorHeight(CTX.arFloorAnchor); + haveFloor = true; + } + } + + for (int32_t i = 0; i < planeCount; i++) { + ArTrackable *trackable = nullptr; + ArTrackableList_acquireItem(CTX.arSession, planes, i, &trackable); + if (trackable == nullptr) { + continue; + } + + ArTrackingState planeState; + ArTrackable_getTrackingState(CTX.arSession, trackable, &planeState); + ArPlaneType planeType; + ArPlane_getType(CTX.arSession, ArAsPlane(trackable), &planeType); + + if (planeState == AR_TRACKING_STATE_TRACKING && + planeType == AR_PLANE_HORIZONTAL_UPWARD_FACING) { + ArPose *planePose = nullptr; + ArPose_create(CTX.arSession, nullptr, &planePose); + ArPlane_getCenterPose(CTX.arSession, ArAsPlane(trackable), planePose); + float planeRaw[7] = {}; + ArPose_getPoseRaw(CTX.arSession, planePose, planeRaw); + float planeY = planeRaw[5]; + + // The floor is the lowest such plane (tables and shelves are higher). + if (!haveFloor || planeY < floorY - ARCORE_FLOOR_REANCHOR_MARGIN_M) { + // Anchor at the plane's position only, without its rotation. (Poses are created + // from {qx, qy, qz, qw, tx, ty, tz}.) + const float translationOnlyRaw[7] = { + 0.f, 0.f, 0.f, 1.f, planeRaw[4], planeRaw[5], planeRaw[6]}; + ArPose *translationOnly = nullptr; + ArPose_create(CTX.arSession, translationOnlyRaw, &translationOnly); + ArAnchor *newAnchor = nullptr; + if (ArTrackable_acquireNewAnchor( + CTX.arSession, trackable, translationOnly, &newAnchor) == AR_SUCCESS) { + if (CTX.arFloorAnchor != nullptr) { + ArAnchor_detach(CTX.arSession, CTX.arFloorAnchor); + ArAnchor_release(CTX.arFloorAnchor); + } + CTX.arFloorAnchor = newAnchor; + info("ARCore: floor %s at y=%.3f", + haveFloor ? "lowered" : "found", + planeY); + floorY = planeY; + haveFloor = true; + } + ArPose_destroy(translationOnly); + } + ArPose_destroy(planePose); + } + ArTrackable_release(trackable); + } + ArTrackableList_destroy(planes); + + if (haveFloor) { + *outFloorY = floorY; + } + return haveFloor; +} + // Updates ARCore once per render frame and publishes what getPose() needs: the ARCore position // and the orientation correction. Must be called from the render thread, with the app's EGL // context current (ArSession_update requires it). @@ -361,6 +480,34 @@ static void updateARCoreTracking() { ArPose_destroy(arPose); ArCamera_release(arCamera); + // Vertical position (see ARCORE_HEIGHT_MODE). + float headY = raw[5]; + float floorY = 0.0f; + bool floorKnown = false; + if (ARCORE_HEIGHT_MODE != 0) { + floorKnown = updateFloorAnchor(&floorY); + } + if (ARCORE_HEIGHT_MODE == 1) { + if (floorKnown) { + headY = raw[5] - floorY; + } + } else if (ARCORE_HEIGHT_MODE == 2) { + if (floorKnown) { + uint64_t blendDtNs = CTX.lastFloorBlendNs != 0 ? nowNs - CTX.lastFloorBlendNs : 0; + if (blendDtNs > 0 && blendDtNs < ARCORE_MAX_CORRECTION_GAP_NS) { + float gain = 1.0f - std::exp(-((float) blendDtNs * 1e-9f) / + ARCORE_FLOOR_BLEND_TIME_CONSTANT_S); + // ARCore's origin is -floorY above the floor. + CTX.floorOffsetM += (-floorY - CTX.floorOffsetM) * gain; + } + CTX.lastFloorBlendNs = nowNs; + } else { + CTX.lastFloorBlendNs = 0; + } + headY = raw[5] + CTX.floorOffsetM; + } + headY += ARCORE_HEIGHT_OFFSET_M; + // How far apart are Cardboard and ARCore right now? AlvrQuat cardboardOrientation = cardboardOrientationAt(nowNs + ESTIMATED_PREDICTION_OFFSET_NS); @@ -379,7 +526,7 @@ static void updateARCoreTracking() { std::lock_guard poseLock(CTX.poseMutex); CTX.arPosition[0] = raw[4]; - CTX.arPosition[1] = raw[5]; + CTX.arPosition[1] = headY; CTX.arPosition[2] = raw[6]; CTX.arPositionValid = true; @@ -407,9 +554,10 @@ static void updateARCoreTracking() { if (shouldLog) { lastLogNs = nowNs; - info("ARCore: tracking, position=(%.3f, %.3f, %.3f) | orientation error=%.2f deg, " - "correction=%.2f deg, sample %s", - raw[4], raw[5], raw[6], errorDeg, correctionDeg, accepted ? "accepted" : "rejected"); + info("ARCore: tracking, position=(%.3f, %.3f, %.3f), floor %s (y=%.2f) | orientation " + "error=%.2f deg, correction=%.2f deg, sample %s", + raw[4], headY, raw[6], floorKnown ? "known" : "unknown", floorY, errorDeg, + correctionDeg, accepted ? "accepted" : "rejected"); } } @@ -544,6 +692,10 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_destro { std::lock_guard lock(CTX.arMutex); + if (CTX.arFloorAnchor != nullptr) { + ArAnchor_release(CTX.arFloorAnchor); + CTX.arFloorAnchor = nullptr; + } if (CTX.arFrame != nullptr) { ArFrame_destroy(CTX.arFrame); CTX.arFrame = nullptr; From 0cc5ee9195184149e7e11283ead7104c144c81b2 Mon Sep 17 00:00:00 2001 From: Rob Oates Date: Wed, 7 Oct 2026 15:51:34 +0900 Subject: [PATCH 13/29] Step 4 and 5: floor height, orientation gate --- .../PhoneVR/app/src/main/cpp/alvr_main.cpp | 71 ++++++++++++++++--- 1 file changed, 61 insertions(+), 10 deletions(-) diff --git a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp index 8678e84f..40772d8b 100644 --- a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp +++ b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp @@ -95,8 +95,15 @@ constexpr bool ARCORE_USE_POSITION = true; constexpr bool ARCORE_USE_ORIENTATION_CORRECTION = true; // How fast the correction follows ARCore. Larger = slower and smoother. constexpr float ARCORE_CORRECTION_TIME_CONSTANT_S = 3.0f; -// If Cardboard and ARCore disagree by more than this, ignore the sample as a bad measurement. -constexpr float ARCORE_MAX_ORIENTATION_ERROR_DEG = 15.0f; +// A sample is ignored as a bad measurement when it disagrees with the orientation we are +// currently showing (Cardboard plus the correction) by more than this. How much drift Cardboard +// has built up in total does not matter, only the disagreement left after the correction. +constexpr float ARCORE_MAX_RESIDUAL_DEG = 15.0f; +// If samples keep disagreeing for this long, ARCore has probably moved its world frame (tracking +// was lost and regained, or it relocalized), so the new value is accepted... +constexpr float ARCORE_REACQUIRE_AFTER_S = 1.5f; +// ...and the correction follows it with this faster time constant until the two agree again. +constexpr float ARCORE_REACQUIRE_TIME_CONSTANT_S = 1.0f; // Ignore the update after a gap longer than this (pause, lifecycle change, debugger). constexpr uint64_t ARCORE_MAX_CORRECTION_GAP_NS = 500000000ULL; // Fixed rotation between ARCore's display-oriented camera frame and Cardboard's frame @@ -111,7 +118,7 @@ const AlvrQuat ARCORE_TO_CARDBOARD = {-0.0057f, -0.0031f, 0.7076f, 0.7066f}; // 2 = like 1, but until a floor is found it assumes the headset started // ARCORE_ASSUMED_START_HEIGHT_M above the floor, and it eases toward the measured floor // height instead of jumping. -constexpr int ARCORE_HEIGHT_MODE = 2; +constexpr int ARCORE_HEIGHT_MODE = 0; // Mode 2 only: assumed height above the floor at the start, before any floor has been detected. constexpr float ARCORE_ASSUMED_START_HEIGHT_M = 1.5f; // Mode 2 only: how slowly the height eases toward the measured floor. Larger = slower. @@ -151,6 +158,7 @@ struct NativeContext { float arPosition[3] = {0.f, 0.f, 0.f}; AlvrQuat orientationCorrection = {0.f, 0.f, 0.f, 1.f}; uint64_t lastCorrectionNs = 0; + uint64_t disagreeSinceNs = 0; // when ARCore started disagreeing with the correction, or 0 int screenWidth = 0; int screenHeight = 0; @@ -517,9 +525,17 @@ static void updateARCoreTracking() { // The rotation that would take Cardboard's orientation onto ARCore's. AlvrQuat relative = normalizeQuat(quatMultiply(quatInverse(cardboardOrientation), arcoreInCardboard)); - float errorDeg = quatAngleDegrees(relative); + // How far Cardboard has drifted from ARCore in total (what the correction has to cancel). + float offsetDeg = quatAngleDegrees(relative); - bool accepted = errorDeg < ARCORE_MAX_ORIENTATION_ERROR_DEG; + static int acceptedCount = 0; + static int rejectedCount = 0; + static float maxResidualDeg = 0.0f; + static bool wasReacquiring = false; + + float residualDeg = 0.0f; + bool accepted = false; + bool reacquiring = false; float correctionDeg = 0.0f; { @@ -530,6 +546,21 @@ static void updateARCoreTracking() { CTX.arPosition[2] = raw[6]; CTX.arPositionValid = true; + // How much ARCore disagrees with what is being shown right now (Cardboard + correction). + AlvrQuat residual = + normalizeQuat(quatMultiply(quatInverse(CTX.orientationCorrection), relative)); + residualDeg = quatAngleDegrees(residual); + + bool agrees = residualDeg < ARCORE_MAX_RESIDUAL_DEG; + if (agrees) { + CTX.disagreeSinceNs = 0; + } else if (CTX.disagreeSinceNs == 0) { + CTX.disagreeSinceNs = nowNs; + } + uint64_t reacquireAfterNs = (uint64_t) (ARCORE_REACQUIRE_AFTER_S * 1e9f); + reacquiring = !agrees && (nowNs - CTX.disagreeSinceNs) >= reacquireAfterNs; + accepted = agrees || reacquiring; + if (accepted) { if (CTX.lastCorrectionNs == 0) { CTX.lastCorrectionNs = nowNs; @@ -538,8 +569,10 @@ static void updateARCoreTracking() { // Protect against an unexpectedly large gap (pause, debugger, lifecycle change). if (deltaNs > 0 && deltaNs < ARCORE_MAX_CORRECTION_GAP_NS) { float dt = (float) deltaNs * 1e-9f; + float timeConstant = + agrees ? ARCORE_CORRECTION_TIME_CONSTANT_S : ARCORE_REACQUIRE_TIME_CONSTANT_S; // Time-based gain, so the result doesn't depend on the frame rate. - float gain = 1.0f - std::exp(-dt / ARCORE_CORRECTION_TIME_CONSTANT_S); + float gain = 1.0f - std::exp(-dt / timeConstant); CTX.orientationCorrection = slerpQuat(CTX.orientationCorrection, relative, gain); } @@ -552,12 +585,30 @@ static void updateARCoreTracking() { correctionDeg = quatAngleDegrees(CTX.orientationCorrection); } + if (accepted) { + acceptedCount++; + } else { + rejectedCount++; + } + maxResidualDeg = std::max(maxResidualDeg, residualDeg); + if (reacquiring && !wasReacquiring) { + info("ARCore: orientation disagreed for %.1f s (residual %.1f deg, offset %.1f deg); " + "following ARCore's new orientation", + ARCORE_REACQUIRE_AFTER_S, residualDeg, offsetDeg); + } + wasReacquiring = reacquiring; + if (shouldLog) { lastLogNs = nowNs; - info("ARCore: tracking, position=(%.3f, %.3f, %.3f), floor %s (y=%.2f) | orientation " - "error=%.2f deg, correction=%.2f deg, sample %s", - raw[4], headY, raw[6], floorKnown ? "known" : "unknown", floorY, errorDeg, - correctionDeg, accepted ? "accepted" : "rejected"); + info("ARCore: tracking, position=(%.3f, %.3f, %.3f), floor %s (y=%.2f) | offset from " + "ARCore=%.2f deg, residual=%.2f deg (max %.2f), correction=%.2f deg, last second: " + "%d accepted, %d rejected%s", + raw[4], headY, raw[6], floorKnown ? "known" : "unknown", floorY, offsetDeg, + residualDeg, maxResidualDeg, correctionDeg, acceptedCount, rejectedCount, + reacquiring ? ", RE-ACQUIRING" : ""); + acceptedCount = 0; + rejectedCount = 0; + maxResidualDeg = 0.0f; } } From 96ce1131b9bfc9c9386c9010c0fe05b3e773d23e Mon Sep 17 00:00:00 2001 From: Rob Oates Date: Thu, 8 Oct 2026 22:11:56 +0900 Subject: [PATCH 14/29] ARCore: floor height, check samples against the residual, world-yaw correction --- .../PhoneVR/app/src/main/cpp/alvr_main.cpp | 114 ++++++++++-------- 1 file changed, 63 insertions(+), 51 deletions(-) diff --git a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp index 40772d8b..d13528ba 100644 --- a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp +++ b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp @@ -110,6 +110,11 @@ constexpr uint64_t ARCORE_MAX_CORRECTION_GAP_NS = 500000000ULL; // (landscape-left). Found by calibration on one phone; other phones may need other values. const AlvrQuat ARCORE_TO_CARDBOARD = {-0.0057f, -0.0031f, 0.7076f, 0.7066f}; +// Which axis is "up" in the orientation this file works with (0 = x, 1 = y, 2 = z). The yaw +// correction is a rotation about this axis. The log prints a "tilt disagreement" that should stay +// small (a few degrees); if it is large, this axis is probably wrong. +constexpr int ARCORE_WORLD_UP_AXIS = 1; + // ---- ARCore head height ---- // How the vertical position is derived: // 0 = relative to where ARCore started (height starts at 0, as in the earlier builds). @@ -156,7 +161,8 @@ struct NativeContext { std::mutex poseMutex; bool arPositionValid = false; float arPosition[3] = {0.f, 0.f, 0.f}; - AlvrQuat orientationCorrection = {0.f, 0.f, 0.f, 1.f}; + // Yaw (rotation about the world's up axis) added to Cardboard's orientation, in radians. + float yawCorrectionRad = 0.f; uint64_t lastCorrectionNs = 0; uint64_t disagreeSinceNs = 0; // when ARCore started disagreeing with the correction, or 0 @@ -271,35 +277,25 @@ static float quatAngleDegrees(const AlvrQuat &q) { return 2.0f * std::acos(w) * 180.0f / (float) M_PI; } -static AlvrQuat slerpQuat(AlvrQuat a, AlvrQuat b, float t) { - a = normalizeQuat(a); - b = normalizeQuat(b); - - float dot = a.x * b.x + a.y * b.y + a.z * b.z + a.w * b.w; - - // Take the shortest path. - if (dot < 0.0f) { - b = {-b.x, -b.y, -b.z, -b.w}; - dot = -dot; +// Wraps an angle to the range [-pi, pi]. +static float wrapRad(float angle) { + const float twoPi = 2.0f * (float) M_PI; + while (angle > (float) M_PI) { + angle -= twoPi; } - - // Very close: normalized linear interpolation is enough. - if (dot > 0.9995f) { - return normalizeQuat({a.x + t * (b.x - a.x), - a.y + t * (b.y - a.y), - a.z + t * (b.z - a.z), - a.w + t * (b.w - a.w)}); + while (angle < -(float) M_PI) { + angle += twoPi; } + return angle; +} - float theta = std::acos(dot); - float sinTheta = std::sin(theta); - float wa = std::sin((1.0f - t) * theta) / sinTheta; - float wb = std::sin(t * theta) / sinTheta; - - return normalizeQuat({wa * a.x + wb * b.x, - wa * a.y + wb * b.y, - wa * a.z + wb * b.z, - wa * a.w + wb * b.w}); +// A rotation about the world's up axis. +static AlvrQuat worldYawQuat(float angleRad) { + float s = std::sin(angleRad * 0.5f); + float c = std::cos(angleRad * 0.5f); + float axis[3] = {0.f, 0.f, 0.f}; + axis[ARCORE_WORLD_UP_AXIS] = s; + return {axis[0], axis[1], axis[2], c}; } // Cardboard's orientation at the given time, in the same convention the rest of this file uses. @@ -312,8 +308,9 @@ static AlvrQuat cardboardOrientationAt(uint64_t timestampNs) { // Head pose, in world/tracking space (not head-relative). Safe to call from any thread. // -// Orientation: Cardboard's fast gyro orientation, multiplied by a slowly changing correction that -// the render thread computes from ARCore (updateARCoreTracking). This is what removes the drift. +// Orientation: Cardboard's fast gyro orientation, turned about the vertical by a slowly changing +// yaw correction that the render thread computes from ARCore (updateARCoreTracking). Gyro drift +// is a turn about the vertical, so this is what removes it. // Position: ARCore's last known position, or the origin until ARCore has tracked (3DoF). AlvrPose getPose(uint64_t timestampNs) { AlvrPose pose = {}; @@ -324,7 +321,8 @@ AlvrPose getPose(uint64_t timestampNs) { std::lock_guard lock(CTX.poseMutex); if (ARCORE_USE_ORIENTATION_CORRECTION) { - orientation = normalizeQuat(quatMultiply(orientation, CTX.orientationCorrection)); + orientation = + normalizeQuat(quatMultiply(worldYawQuat(CTX.yawCorrectionRad), orientation)); } if (ARCORE_USE_POSITION && CTX.arPositionValid) { @@ -522,15 +520,29 @@ static void updateARCoreTracking() { AlvrQuat arcoreOrientation = normalizeQuat(AlvrQuat{raw[0], raw[1], raw[2], raw[3]}); AlvrQuat arcoreInCardboard = normalizeQuat(quatMultiply(arcoreOrientation, quatInverse(ARCORE_TO_CARDBOARD))); - // The rotation that would take Cardboard's orientation onto ARCore's. - AlvrQuat relative = - normalizeQuat(quatMultiply(quatInverse(cardboardOrientation), arcoreInCardboard)); - // How far Cardboard has drifted from ARCore in total (what the correction has to cancel). - float offsetDeg = quatAngleDegrees(relative); + + // The rotation, in world space, that takes Cardboard's orientation onto ARCore's. Both sensors + // know which way is down, so this is (nearly) a pure rotation about the vertical: the yaw + // drift. Splitting off that yaw (and treating what is left as "tilt") makes the correction + // independent of where the head is pointing. A correction expressed in the phone's own frame + // is only right while the head keeps the pose it was measured in. + AlvrQuat worldOffset = + normalizeQuat(quatMultiply(arcoreInCardboard, quatInverse(cardboardOrientation))); + if (worldOffset.w < 0.0f) { + worldOffset = {-worldOffset.x, -worldOffset.y, -worldOffset.z, -worldOffset.w}; + } + const float offsetComponents[3] = {worldOffset.x, worldOffset.y, worldOffset.z}; + float measuredYawRad = + 2.0f * std::atan2(offsetComponents[ARCORE_WORLD_UP_AXIS], worldOffset.w); + AlvrQuat tilt = normalizeQuat( + quatMultiply(quatInverse(worldYawQuat(measuredYawRad)), worldOffset)); + float tiltDeg = quatAngleDegrees(tilt); + float yawOffsetDeg = measuredYawRad * 180.0f / (float) M_PI; static int acceptedCount = 0; static int rejectedCount = 0; static float maxResidualDeg = 0.0f; + static float maxTiltDeg = 0.0f; static bool wasReacquiring = false; float residualDeg = 0.0f; @@ -546,10 +558,9 @@ static void updateARCoreTracking() { CTX.arPosition[2] = raw[6]; CTX.arPositionValid = true; - // How much ARCore disagrees with what is being shown right now (Cardboard + correction). - AlvrQuat residual = - normalizeQuat(quatMultiply(quatInverse(CTX.orientationCorrection), relative)); - residualDeg = quatAngleDegrees(residual); + // How much ARCore disagrees with the yaw being applied right now. + float residualRad = wrapRad(measuredYawRad - CTX.yawCorrectionRad); + residualDeg = std::abs(residualRad) * 180.0f / (float) M_PI; bool agrees = residualDeg < ARCORE_MAX_RESIDUAL_DEG; if (agrees) { @@ -573,8 +584,7 @@ static void updateARCoreTracking() { agrees ? ARCORE_CORRECTION_TIME_CONSTANT_S : ARCORE_REACQUIRE_TIME_CONSTANT_S; // Time-based gain, so the result doesn't depend on the frame rate. float gain = 1.0f - std::exp(-dt / timeConstant); - CTX.orientationCorrection = - slerpQuat(CTX.orientationCorrection, relative, gain); + CTX.yawCorrectionRad = wrapRad(CTX.yawCorrectionRad + residualRad * gain); } CTX.lastCorrectionNs = nowNs; } else { @@ -582,7 +592,7 @@ static void updateARCoreTracking() { CTX.lastCorrectionNs = nowNs; } - correctionDeg = quatAngleDegrees(CTX.orientationCorrection); + correctionDeg = CTX.yawCorrectionRad * 180.0f / (float) M_PI; } if (accepted) { @@ -591,24 +601,26 @@ static void updateARCoreTracking() { rejectedCount++; } maxResidualDeg = std::max(maxResidualDeg, residualDeg); + maxTiltDeg = std::max(maxTiltDeg, tiltDeg); if (reacquiring && !wasReacquiring) { - info("ARCore: orientation disagreed for %.1f s (residual %.1f deg, offset %.1f deg); " - "following ARCore's new orientation", - ARCORE_REACQUIRE_AFTER_S, residualDeg, offsetDeg); + info("ARCore: yaw disagreed for %.1f s (residual %.1f deg, offset %.1f deg); following " + "ARCore's new yaw", + ARCORE_REACQUIRE_AFTER_S, residualDeg, yawOffsetDeg); } wasReacquiring = reacquiring; if (shouldLog) { lastLogNs = nowNs; - info("ARCore: tracking, position=(%.3f, %.3f, %.3f), floor %s (y=%.2f) | offset from " - "ARCore=%.2f deg, residual=%.2f deg (max %.2f), correction=%.2f deg, last second: " - "%d accepted, %d rejected%s", - raw[4], headY, raw[6], floorKnown ? "known" : "unknown", floorY, offsetDeg, - residualDeg, maxResidualDeg, correctionDeg, acceptedCount, rejectedCount, - reacquiring ? ", RE-ACQUIRING" : ""); + info("ARCore: tracking, position=(%.3f, %.3f, %.3f), floor %s (y=%.2f) | yaw offset from " + "ARCore=%.2f deg, residual=%.2f deg (max %.2f), yaw correction=%.2f deg, tilt " + "disagreement=%.2f deg (max %.2f), last second: %d accepted, %d rejected%s", + raw[4], headY, raw[6], floorKnown ? "known" : "unknown", floorY, yawOffsetDeg, + residualDeg, maxResidualDeg, correctionDeg, tiltDeg, maxTiltDeg, acceptedCount, + rejectedCount, reacquiring ? ", RE-ACQUIRING" : ""); acceptedCount = 0; rejectedCount = 0; maxResidualDeg = 0.0f; + maxTiltDeg = 0.0f; } } From e01a84a074893240fbaad70c335d9840d87998b1 Mon Sep 17 00:00:00 2001 From: Rob Oates Date: Sat, 10 Oct 2026 12:04:31 +0900 Subject: [PATCH 15/29] Start ARCore after video starts flowing (fixes frozen stream on some phones) --- .../PhoneVR/app/src/main/cpp/alvr_main.cpp | 94 +++++++++++-- notes/ARCORE_NOTES.md | 130 ++++++++++++++++++ .../MIGRATION_NOTES.md | 0 3 files changed, 209 insertions(+), 15 deletions(-) create mode 100644 notes/ARCORE_NOTES.md rename MIGRATION_NOTES.md => notes/MIGRATION_NOTES.md (100%) diff --git a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp index d13528ba..0262bf5a 100644 --- a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp +++ b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp @@ -90,6 +90,13 @@ const int MAXIMUM_TRACKING_FRAMES = 360; // ---- ARCore tuning (values carried over from the v20 branch; tuned on one phone only) ---- // Use ARCore's position as the head position (otherwise position stays at the origin: 3DoF). +// Some phones (seen on an AQUOS sense4 basic, Adreno 618, Android 12) stop delivering decoded +// video frames while the ARCore camera session is running. With this set, ARCore is only +// started once the video decoder has produced a few frames, and it is paused again when the +// stream stops. Set to false to start ARCore together with the app, as before. +constexpr bool ARCORE_START_AFTER_VIDEO = true; +constexpr int ARCORE_START_AFTER_VIDEO_FRAMES = 10; + constexpr bool ARCORE_USE_POSITION = true; // Slowly pull Cardboard's drifting orientation toward ARCore's (complementary filter). constexpr bool ARCORE_USE_ORIENTATION_CORRECTION = true; @@ -145,6 +152,8 @@ struct NativeContext { // concurrently, so every use of arSession is guarded by arMutex. ArSession_update() must // additionally run on the render thread, with the app's EGL context current. bool arcoreEnabled = false; + bool arSessionRunning = false; // ArSession_resume() done and not yet paused; under arMutex + int videoFramesSeen = 0; // decoded frames since the stream started (render thread) ArSession *arSession = nullptr; ArFrame *arFrame = nullptr; // Anchor on the lowest horizontal plane found so far (taken to be the floor). An anchor keeps @@ -434,6 +443,38 @@ static bool updateFloorAnchor(float *outFloorY) { // Updates ARCore once per render frame and publishes what getPose() needs: the ARCore position // and the orientation correction. Must be called from the render thread, with the app's EGL // context current (ArSession_update requires it). +// Starts the ARCore camera session (no-op if it is already running or ARCore is off). +static void startARCoreSession(const char *why) { + std::lock_guard lock(CTX.arMutex); + if (!CTX.arcoreEnabled || CTX.arSession == nullptr || CTX.arSessionRunning) { + return; + } + ArStatus arStatus = ArSession_resume(CTX.arSession); + if (arStatus != AR_SUCCESS) { + info("ARCore ERROR: failed to resume session (status %d)", (int) arStatus); + return; + } + CTX.arSessionRunning = true; + info("ARCore: camera session started (%s)", why); +} + +// Pauses the ARCore session, which releases the camera (no-op if it is not running). +static void stopARCoreSession(const char *why) { + { + std::lock_guard lock(CTX.arMutex); + if (!CTX.arcoreEnabled || CTX.arSession == nullptr || !CTX.arSessionRunning) { + return; + } + ArSession_pause(CTX.arSession); + CTX.arSessionRunning = false; + } + { + std::lock_guard lock(CTX.poseMutex); + CTX.arPositionValid = false; + } + info("ARCore: camera session paused (%s)", why); +} + static void updateARCoreTracking() { if (!CTX.arcoreEnabled || CTX.arTexture == 0) { return; @@ -444,7 +485,7 @@ static void updateARCoreTracking() { } std::lock_guard lock(CTX.arMutex); - if (CTX.arSession == nullptr || CTX.arFrame == nullptr) { + if (CTX.arSession == nullptr || CTX.arFrame == nullptr || !CTX.arSessionRunning) { return; } @@ -775,14 +816,8 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_resume jobject) { CardboardHeadTracker_resume(CTX.headTracker); - { - std::lock_guard lock(CTX.arMutex); - if (CTX.arcoreEnabled && CTX.arSession != nullptr) { - ArStatus arStatus = ArSession_resume(CTX.arSession); - if (arStatus != AR_SUCCESS) { - info("ARCore ERROR: failed to resume session (status %d)", (int) arStatus); - } - } + if (!ARCORE_START_AFTER_VIDEO) { + startARCoreSession("app resumed"); } CTX.renderingParamsChanged = true; @@ -814,12 +849,7 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_pauseN CardboardHeadTracker_pause(CTX.headTracker); // Releases the camera. (The v20 code never paused the ARCore session.) - { - std::lock_guard lock(CTX.arMutex); - if (CTX.arcoreEnabled && CTX.arSession != nullptr) { - ArSession_pause(CTX.arSession); - } - } + stopARCoreSession("app paused"); } extern "C" JNIEXPORT void JNICALL @@ -1025,6 +1055,7 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render // the v20 code did). CTX.hasFrameMetadata = false; CTX.lastStreamBuffer = nullptr; + CTX.videoFramesSeen = 0; GL(glGenTextures(2, CTX.streamTextures)); @@ -1152,6 +1183,11 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render CTX.streaming = false; CTX.inputThread.join(); + // The camera must be off before the next decoder is created. + if (ARCORE_START_AFTER_VIDEO) { + stopARCoreSession("stream stopped"); + } + // The saved frame belongs to the decoder that is about to be destroyed. CTX.lastStreamBuffer = nullptr; if (g_decoderCreated) { @@ -1190,8 +1226,17 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render // below. bool isNewFrame = alvr_get_frame(×tampNs, &streamHardwareBuffer); + // DIAGNOSTIC (phone-1 freeze): count what the stream branch does, log once a second. + static uint64_t diagLastNs = 0, diagCalls = 0, diagNew = 0, diagNoMeta = 0, diagRendered = 0; + diagCalls++; + if (isNewFrame) diagNew++; + AlvrVideoFrameMetadata metadata; if (isNewFrame) { + if (ARCORE_START_AFTER_VIDEO && CTX.videoFramesSeen < ARCORE_START_AFTER_VIDEO_FRAMES && + ++CTX.videoFramesSeen >= ARCORE_START_AFTER_VIDEO_FRAMES) { + startARCoreSession("video is flowing"); + } CTX.lastStreamBuffer = streamHardwareBuffer; CTX.lastStreamTimestampNs = timestampNs; @@ -1206,6 +1251,7 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render metadata = CTX.lastFrameMetadata; } else { // No metadata at all yet (first frame) - nothing sensible to reproject against. + diagNoMeta++; return; } } else if (CTX.lastStreamBuffer != nullptr && CTX.hasFrameMetadata) { @@ -1215,6 +1261,12 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render metadata = CTX.lastFrameMetadata; } else { // Nothing has been decoded yet. + { + static uint64_t lastNs = 0; uint64_t n = GetBootTimeNano(); + if (n - lastNs > 1000000000ULL) { lastNs = n; + __android_log_print(ANDROID_LOG_INFO, "ALVR_PVR_NATIVE", + "DIAG stream: no decoded frame yet (alvr_get_frame returned false, calls=%llu)", (unsigned long long)diagCalls); } + } return; } @@ -1240,6 +1292,18 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render ALVR_GL_CALL( alvr_render_stream_opengl(streamHardwareBuffer, streamViewParams, foveationCenters)); EGLCHK("after alvr_render_stream_opengl()"); + diagRendered++; + { + uint64_t nowNs = GetBootTimeNano(); + if (nowNs - diagLastNs > 1000000000ULL) { + __android_log_print(ANDROID_LOG_INFO, "ALVR_PVR_NATIVE", + "DIAG stream: calls=%llu newFrames=%llu noMetadataReturns=%llu rendered=%llu lastTs=%llu glError=0x%x", + (unsigned long long)diagCalls, (unsigned long long)diagNew, + (unsigned long long)diagNoMeta, (unsigned long long)diagRendered, + (unsigned long long)timestampNs, glGetError()); + diagLastNs = nowNs; diagCalls = diagNew = diagNoMeta = diagRendered = 0; + } + } // The statistics are per decoded frame, so only report frames that are new. if (isNewFrame) { diff --git a/notes/ARCORE_NOTES.md b/notes/ARCORE_NOTES.md new file mode 100644 index 00000000..43a804b2 --- /dev/null +++ b/notes/ARCORE_NOTES.md @@ -0,0 +1,130 @@ +# ARCore 6DoF on the v21 client + +Last updated: 2026-10-07 + +## Status + +ARCore position tracking and drift-free orientation work on the ALVR v21 client (branch +`arcore-on-v21`). After the world-yaw correction (step 6 below), a play session of roughly half an +hour showed no drift and no swimming. Everything was tested on one phone only. + +Orientation comes from the Cardboard head tracker (fast, but its yaw drifts). ARCore supplies the +position and a slow yaw correction that cancels the drift. Everything lives in +`app/src/main/cpp/alvr_main.cpp`. + +## How it fits together + +- **Startup:** `initializeNative(..., enableARCore)` creates and configures the ARCore session + (depth, light estimation and cloud anchors off; horizontal and vertical plane finding on; update + mode "latest camera image", so `ArSession_update` does not block). If creation fails, the app + continues as plain 3DoF. +- **Lifecycle:** the session is resumed in `resumeNative`, paused in `pauseNative` and destroyed in + `destroyNative`. Every ARCore call is guarded by `CTX.arMutex`. +- **Camera texture:** ARCore needs a GL texture to write camera frames into. One is created right + after `alvr_resume_opengl`. It is a plain `GL_TEXTURE_2D`; ARCore's documentation asks for an + external-OES texture, but tracking worked this way on v20 and v21, so it was left alone. +- **Once per render frame:** `updateARCoreTracking()` runs on the render thread after the event + loop. `ArSession_update` must run there, with the app's EGL context current. It reads ARCore's + camera pose, publishes the position and updates the yaw correction. +- **From any thread:** `getPose(timestampNs)` returns Cardboard's orientation at that time, turned + about the vertical by the current yaw correction, plus ARCore's last known position. The input + thread, the render thread and the lobby all use it, so everything sees the same orientation. + +## The yaw correction (complementary filter) + +Each frame the code computes the world-space rotation that takes Cardboard's orientation onto +ARCore's. Both sensors know which way is down, so this is (nearly) a pure rotation about the +vertical, which is the yaw drift. The yaw part is extracted and a slow filter follows it: + +- a sample is used only if it disagrees with the yaw currently being applied by less than + `ARCORE_MAX_RESIDUAL_DEG` +- the correction follows ARCore with time constant `ARCORE_CORRECTION_TIME_CONSTANT_S`, using a + gain that depends on elapsed time, so it does not depend on the frame rate +- if samples keep disagreeing for `ARCORE_REACQUIRE_AFTER_S`, ARCore has probably moved its own + world frame (tracking lost and regained, or a relocalization), so the filter follows the new + value with the faster `ARCORE_REACQUIRE_TIME_CONSTANT_S` until they agree again +- gaps longer than `ARCORE_MAX_CORRECTION_GAP_NS` (pause, debugger, lifecycle change) are skipped + +### Why it is built this way + +The first port copied the v20 filter, which had two flaws that only show up in long sessions: + +1. **The disagreement check compared ARCore with raw Cardboard**, so it measured the total drift + built up, not the disagreement left after correction. Once Cardboard had drifted about 15 degrees + the filter rejected every sample and froze, and yaw drift appeared after five to six minutes. + The check now compares against the orientation being shown. +2. **The correction was a rotation in the phone's own frame.** Gyro drift is a turn about the + vertical, so a phone-frame correction is only right while the head keeps the pose it was + measured in. Once the correction was large, looking up and down made the view roll and swim. + A 21-minute log showed the disagreement growing with the size of the correction: about 1 degree + below 20 degrees of correction, 3 degrees at 20 to 55, and about 8 degrees (with 12 percent of + samples rejected) above 55. The correction is now a single angle about the vertical, which does + not depend on where the head points. + +## Tuning + +All values are constants near the top of `alvr_main.cpp`. They were tuned on one phone. + +| Constant | Meaning | +|---|---| +| `ARCORE_USE_POSITION` | Use ARCore's position as the head position. | +| `ARCORE_USE_ORIENTATION_CORRECTION` | Apply the yaw correction. | +| `ARCORE_CORRECTION_TIME_CONSTANT_S` | How fast the correction follows ARCore. Larger is smoother. | +| `ARCORE_MAX_RESIDUAL_DEG` | Disagreement above which a sample is ignored. | +| `ARCORE_REACQUIRE_AFTER_S`, `ARCORE_REACQUIRE_TIME_CONSTANT_S` | Re-acquire behaviour described above. | +| `ARCORE_MAX_CORRECTION_GAP_NS` | Longest time gap that is still filtered. | +| `ARCORE_TO_CARDBOARD` | Fixed rotation between ARCore's display-oriented camera frame and Cardboard's frame (landscape-left). Found by calibration on one phone. **Other phones may need a different value.** | +| `ARCORE_WORLD_UP_AXIS` | Which axis is up in the orientation the code uses (default y). | + +## Height + +`ARCORE_HEIGHT_MODE` selects how the vertical position is derived: + +- **0:** relative to where ARCore started. +- **1:** height above the detected floor (the v20 behaviour). Until a floor is found it is the same + as 0, so the height jumps when one is first detected. +- **2 (default):** assumes `ARCORE_ASSUMED_START_HEIGHT_M` above the floor until one is found, then + eases toward the measured height (`ARCORE_FLOOR_BLEND_TIME_CONSTANT_S`). + +`ARCORE_HEIGHT_OFFSET_M` is added in every mode, for tuning by hand. The floor is the lowest +horizontal, upward-facing plane ARCore reports, held by an anchor so it follows ARCore's frame. +**In a 21-minute session with the phone in the headset, no floor was ever found**, because the +camera rarely sees the floor. Do not rely on floor detection. Which mode and offset suit a given +setup is not settled. + +## Related display fix (stuttering) + +When the decoder has no new frame on a display refresh, the client draws the previous frame again +with the current head rotation, instead of returning without drawing. The saved buffer is cleared +whenever a stream starts or stops or the decoder is created or destroyed, because the buffer +belongs to the decoder. + +## Build and device requirements + +- ARCore 1.46.0 from Maven; Gradle also unpacks its native libraries (`natives` configuration). +- The ARCore C headers are downloaded by `prepare-alvr-deps.sh` from the `main` branch of + `google-ar/arcore-android-sdk`. **They should be pinned to a tag that matches the library + version.** +- The camera permission is already in the manifest. The manifest has no `com.google.ar.core` + `meta-data` entry (`optional` or `required`); add it before releasing, so other phones can + install or update ARCore. +- The phone needs Google Play Services for AR. + +## Reading the log + +`updateARCoreTracking()` prints one line a second: position, floor status, yaw offset from ARCore, +residual (and the largest residual that second), the applied yaw correction, the tilt disagreement +(the part of the difference that is not yaw; it should stay small, and if it is consistently large +the up axis is probably wrong), samples accepted and rejected, and `RE-ACQUIRING` when the filter is +following a new ARCore yaw. Filter with +`adb logcat -d | grep -E "ARCore: (tracking|not tracking|floor|yaw)"`. + +## Not done / ideas + +- Per-phone calibration of `ARCORE_TO_CARDBOARD` and the other constants, as a setting. +- Try a newer ARCore as one isolated change. +- A speed gate (ignore samples during fast head turns, when timing errors are largest). +- Snap the correction at the first tracked frame instead of easing in from zero. +- Attribution: the first ARCore plumbing came from serialuart's experimental PhoneVR 6DoF branch + https://github.com/serialuart/PhoneVR/tree/arcore-workingtree. +- ARCore plane finding is enabled for both horizontal and vertical planes; only horizontal is used. diff --git a/MIGRATION_NOTES.md b/notes/MIGRATION_NOTES.md similarity index 100% rename from MIGRATION_NOTES.md rename to notes/MIGRATION_NOTES.md From 20a85ad92c88708c4195f5f9b51d14db33083d6e Mon Sep 17 00:00:00 2001 From: Rob Oates Date: Sat, 10 Oct 2026 20:05:40 +0900 Subject: [PATCH 16/29] wgpu-hal patch: share EGL context; disable GL debug output (fixes PowerVR GE8300 crash) --- .../wgpu-hal-25.0.2-share-egl-context.patch | 1606 ++++++++++++++++- 1 file changed, 1592 insertions(+), 14 deletions(-) diff --git a/code/mobile/android/PhoneVR/patches/wgpu-hal-25.0.2-share-egl-context.patch b/code/mobile/android/PhoneVR/patches/wgpu-hal-25.0.2-share-egl-context.patch index ffb250f9..10505be4 100644 --- a/code/mobile/android/PhoneVR/patches/wgpu-hal-25.0.2-share-egl-context.patch +++ b/code/mobile/android/PhoneVR/patches/wgpu-hal-25.0.2-share-egl-context.patch @@ -1,12 +1,6 @@ -diff -ruN -x '*.bak' -x target a/.cargo-ok b/.cargo-ok ---- a/.cargo-ok 1970-01-01 09:00:00.000000000 +0900 -+++ b/.cargo-ok 2026-10-05 01:32:32.592471619 +0900 -@@ -0,0 +1 @@ -+{"v":1} -\ No newline at end of file -diff -ruN -x '*.bak' -x target a/src/gles/egl.rs b/src/gles/egl.rs ---- a/src/gles/egl.rs 2026-10-05 01:32:32.518542445 +0900 -+++ b/src/gles/egl.rs 2026-10-05 01:32:32.561883826 +0900 +diff -ruN a/src/gles/egl.rs b/src/gles/egl.rs +--- a/src/gles/egl.rs 2026-10-10 19:59:35.659395136 +0900 ++++ b/src/gles/egl.rs 2026-10-10 19:59:35.673210214 +0900 @@ -410,9 +410,13 @@ impl<'a> Drop for AdapterContextLock<'a> { fn drop(&mut self) { @@ -31,7 +25,17 @@ diff -ruN -x '*.bak' -x target a/src/gles/egl.rs b/src/gles/egl.rs EglContextLock { instance: &egl.instance, display: egl.display, -@@ -654,11 +660,16 @@ +@@ -615,7 +621,8 @@ + wgt::Gles3MinorVersion::Version2 => 2, + }); + } +- if flags.contains(wgt::InstanceFlags::DEBUG) { ++ // PHONEVR: no debug EGL context either (same PowerVR driver bug). ++ if false && flags.contains(wgt::InstanceFlags::DEBUG) { + if version >= (1, 5) { + log::debug!("\tEGL context: +debug"); + context_attributes.push(khronos_egl::CONTEXT_OPENGL_DEBUG); +@@ -654,11 +661,16 @@ gl_context_attributes.extend(&context_attributes); gles_context_attributes.extend(&context_attributes); @@ -51,7 +55,7 @@ diff -ruN -x '*.bak' -x target a/src/gles/egl.rs b/src/gles/egl.rs }) .map_err(|e| { crate::InstanceError::with_source( -@@ -667,7 +678,7 @@ +@@ -667,7 +679,7 @@ ) }) } else { @@ -60,9 +64,1583 @@ diff -ruN -x '*.bak' -x target a/src/gles/egl.rs b/src/gles/egl.rs .map_err(|e| { crate::InstanceError::with_source( String::from("unable to create GLES 3.x context"), -diff -ruN -x '*.bak' -x target a/src/gles/queue.rs b/src/gles/queue.rs ---- a/src/gles/queue.rs 2026-10-05 01:32:32.518643908 +0900 -+++ b/src/gles/queue.rs 2026-10-05 01:32:32.568580204 +0900 +@@ -1099,7 +1111,9 @@ + }); + } + +- if self.flags.contains(wgt::InstanceFlags::VALIDATION) && gl.supports_debug() { ++ // PHONEVR: GL debug output stays off; the PowerVR GE8300 driver crashes while formatting ++ // one of its own debug messages inside glTexSubImage2D (SIGSEGV in strlen/vsnprintf). ++ if false && self.flags.contains(wgt::InstanceFlags::VALIDATION) && gl.supports_debug() { + log::debug!("Enabling GLES debug output"); + unsafe { gl.enable(glow::DEBUG_OUTPUT) }; + unsafe { gl.debug_message_callback(super::gl_debug_message_callback) }; +diff -ruN a/src/gles/egl.rs.bak b/src/gles/egl.rs.bak +--- a/src/gles/egl.rs.bak 1970-01-01 09:00:00.000000000 +0900 ++++ b/src/gles/egl.rs.bak 2026-10-10 19:59:35.674178122 +0900 +@@ -0,0 +1,1559 @@ ++#![allow(clippy::std_instead_of_alloc, clippy::std_instead_of_core)] ++ ++use std::{ ++ ffi, ++ mem::ManuallyDrop, ++ os::raw, ++ ptr, ++ rc::Rc, ++ string::String, ++ sync::{Arc, LazyLock}, ++ time::Duration, ++ vec::Vec, ++}; ++ ++use glow::HasContext; ++use hashbrown::HashMap; ++use parking_lot::{MappedMutexGuard, Mutex, MutexGuard, RwLock}; ++ ++/// The amount of time to wait while trying to obtain a lock to the adapter context ++const CONTEXT_LOCK_TIMEOUT_SECS: u64 = 1; ++ ++const EGL_CONTEXT_FLAGS_KHR: i32 = 0x30FC; ++const EGL_CONTEXT_OPENGL_DEBUG_BIT_KHR: i32 = 0x0001; ++const EGL_CONTEXT_OPENGL_ROBUST_ACCESS_EXT: i32 = 0x30BF; ++const EGL_PLATFORM_WAYLAND_KHR: u32 = 0x31D8; ++const EGL_PLATFORM_X11_KHR: u32 = 0x31D5; ++const EGL_PLATFORM_ANGLE_ANGLE: u32 = 0x3202; ++const EGL_PLATFORM_ANGLE_NATIVE_PLATFORM_TYPE_ANGLE: u32 = 0x348F; ++const EGL_PLATFORM_ANGLE_DEBUG_LAYERS_ENABLED: u32 = 0x3451; ++const EGL_PLATFORM_SURFACELESS_MESA: u32 = 0x31DD; ++const EGL_GL_COLORSPACE_KHR: u32 = 0x309D; ++const EGL_GL_COLORSPACE_SRGB_KHR: u32 = 0x3089; ++ ++type XOpenDisplayFun = ++ unsafe extern "system" fn(display_name: *const raw::c_char) -> *mut raw::c_void; ++ ++type XCloseDisplayFun = unsafe extern "system" fn(display: *mut raw::c_void) -> raw::c_int; ++ ++type WlDisplayConnectFun = ++ unsafe extern "system" fn(display_name: *const raw::c_char) -> *mut raw::c_void; ++ ++type WlDisplayDisconnectFun = unsafe extern "system" fn(display: *const raw::c_void); ++ ++#[cfg(not(Emscripten))] ++type EglInstance = khronos_egl::DynamicInstance; ++ ++#[cfg(Emscripten)] ++type EglInstance = khronos_egl::Instance; ++ ++type WlEglWindowCreateFun = unsafe extern "system" fn( ++ surface: *const raw::c_void, ++ width: raw::c_int, ++ height: raw::c_int, ++) -> *mut raw::c_void; ++ ++type WlEglWindowResizeFun = unsafe extern "system" fn( ++ window: *const raw::c_void, ++ width: raw::c_int, ++ height: raw::c_int, ++ dx: raw::c_int, ++ dy: raw::c_int, ++); ++ ++type WlEglWindowDestroyFun = unsafe extern "system" fn(window: *const raw::c_void); ++ ++type EglLabel = *const raw::c_void; ++ ++#[allow(clippy::upper_case_acronyms)] ++type EGLDEBUGPROCKHR = Option< ++ unsafe extern "system" fn( ++ error: khronos_egl::Enum, ++ command: *const raw::c_char, ++ message_type: u32, ++ thread_label: EglLabel, ++ object_label: EglLabel, ++ message: *const raw::c_char, ++ ), ++>; ++ ++const EGL_DEBUG_MSG_CRITICAL_KHR: u32 = 0x33B9; ++const EGL_DEBUG_MSG_ERROR_KHR: u32 = 0x33BA; ++const EGL_DEBUG_MSG_WARN_KHR: u32 = 0x33BB; ++const EGL_DEBUG_MSG_INFO_KHR: u32 = 0x33BC; ++ ++type EglDebugMessageControlFun = unsafe extern "system" fn( ++ proc: EGLDEBUGPROCKHR, ++ attrib_list: *const khronos_egl::Attrib, ++) -> raw::c_int; ++ ++unsafe extern "system" fn egl_debug_proc( ++ error: khronos_egl::Enum, ++ command_raw: *const raw::c_char, ++ message_type: u32, ++ _thread_label: EglLabel, ++ _object_label: EglLabel, ++ message_raw: *const raw::c_char, ++) { ++ let log_severity = match message_type { ++ EGL_DEBUG_MSG_CRITICAL_KHR | EGL_DEBUG_MSG_ERROR_KHR => log::Level::Error, ++ EGL_DEBUG_MSG_WARN_KHR => log::Level::Warn, ++ EGL_DEBUG_MSG_INFO_KHR => log::Level::Info, ++ _ => log::Level::Debug, ++ }; ++ let command = unsafe { ffi::CStr::from_ptr(command_raw) }.to_string_lossy(); ++ let message = if message_raw.is_null() { ++ "".into() ++ } else { ++ unsafe { ffi::CStr::from_ptr(message_raw) }.to_string_lossy() ++ }; ++ ++ log::log!( ++ log_severity, ++ "EGL '{}' code 0x{:x}: {}", ++ command, ++ error, ++ message, ++ ); ++} ++ ++/// A simple wrapper around an X11 or Wayland display handle. ++/// Since the logic in this file doesn't actually need to directly ++/// persist a wayland connection handle, the only load-bearing ++/// enum variant is the X11 variant ++#[derive(Debug)] ++enum DisplayRef { ++ X11(ptr::NonNull), ++ Wayland, ++} ++ ++impl DisplayRef { ++ /// Convenience for getting the underlying pointer ++ fn as_ptr(&self) -> *mut raw::c_void { ++ match *self { ++ Self::X11(ptr) => ptr.as_ptr(), ++ Self::Wayland => unreachable!(), ++ } ++ } ++} ++ ++/// DisplayOwner ties the lifetime of the system display handle ++/// to that of the loaded library. ++/// It implements Drop to ensure that the display handle is closed ++/// prior to unloading the library so that we don't leak the ++/// associated file descriptors ++#[derive(Debug)] ++struct DisplayOwner { ++ library: libloading::Library, ++ display: DisplayRef, ++} ++ ++impl Drop for DisplayOwner { ++ fn drop(&mut self) { ++ match self.display { ++ DisplayRef::X11(ptr) => unsafe { ++ let func: libloading::Symbol = ++ self.library.get(c"XCloseDisplay".to_bytes()).unwrap(); ++ func(ptr.as_ptr()); ++ }, ++ DisplayRef::Wayland => {} ++ } ++ } ++} ++ ++fn open_x_display() -> Option { ++ log::debug!("Loading X11 library to get the current display"); ++ unsafe { ++ let library = find_library(&["libX11.so.6", "libX11.so"])?; ++ let func: libloading::Symbol = ++ library.get(c"XOpenDisplay".to_bytes()).unwrap(); ++ let result = func(ptr::null()); ++ ptr::NonNull::new(result).map(|ptr| DisplayOwner { ++ display: DisplayRef::X11(ptr), ++ library, ++ }) ++ } ++} ++ ++unsafe fn find_library(paths: &[&str]) -> Option { ++ for path in paths { ++ match unsafe { libloading::Library::new(path) } { ++ Ok(lib) => return Some(lib), ++ _ => continue, ++ }; ++ } ++ None ++} ++ ++fn test_wayland_display() -> Option { ++ /* We try to connect and disconnect here to simply ensure there ++ * is an active wayland display available. ++ */ ++ log::debug!("Loading Wayland library to get the current display"); ++ let library = unsafe { ++ let client_library = find_library(&["libwayland-client.so.0", "libwayland-client.so"])?; ++ let wl_display_connect: libloading::Symbol = client_library ++ .get(c"wl_display_connect".to_bytes()) ++ .unwrap(); ++ let wl_display_disconnect: libloading::Symbol = client_library ++ .get(c"wl_display_disconnect".to_bytes()) ++ .unwrap(); ++ let display = ptr::NonNull::new(wl_display_connect(ptr::null()))?; ++ wl_display_disconnect(display.as_ptr()); ++ find_library(&["libwayland-egl.so.1", "libwayland-egl.so"])? ++ }; ++ Some(DisplayOwner { ++ library, ++ display: DisplayRef::Wayland, ++ }) ++} ++ ++#[derive(Clone, Copy, Debug)] ++enum SrgbFrameBufferKind { ++ /// No support for SRGB surface ++ None, ++ /// Using EGL 1.5's support for colorspaces ++ Core, ++ /// Using EGL_KHR_gl_colorspace ++ Khr, ++} ++ ++/// Choose GLES framebuffer configuration. ++fn choose_config( ++ egl: &EglInstance, ++ display: khronos_egl::Display, ++ srgb_kind: SrgbFrameBufferKind, ++) -> Result<(khronos_egl::Config, bool), crate::InstanceError> { ++ //TODO: EGL_SLOW_CONFIG ++ let tiers = [ ++ ( ++ "off-screen", ++ &[ ++ khronos_egl::SURFACE_TYPE, ++ khronos_egl::PBUFFER_BIT, ++ khronos_egl::RENDERABLE_TYPE, ++ khronos_egl::OPENGL_ES2_BIT, ++ ][..], ++ ), ++ ( ++ "presentation", ++ &[khronos_egl::SURFACE_TYPE, khronos_egl::WINDOW_BIT][..], ++ ), ++ #[cfg(not(target_os = "android"))] ++ ( ++ "native-render", ++ &[khronos_egl::NATIVE_RENDERABLE, khronos_egl::TRUE as _][..], ++ ), ++ ]; ++ ++ let mut attributes = Vec::with_capacity(9); ++ for tier_max in (0..tiers.len()).rev() { ++ let name = tiers[tier_max].0; ++ log::debug!("\tTrying {}", name); ++ ++ attributes.clear(); ++ for &(_, tier_attr) in tiers[..=tier_max].iter() { ++ attributes.extend_from_slice(tier_attr); ++ } ++ // make sure the Alpha is enough to support sRGB ++ match srgb_kind { ++ SrgbFrameBufferKind::None => {} ++ _ => { ++ attributes.push(khronos_egl::ALPHA_SIZE); ++ attributes.push(8); ++ } ++ } ++ attributes.push(khronos_egl::NONE); ++ ++ match egl.choose_first_config(display, &attributes) { ++ Ok(Some(config)) => { ++ if tier_max == 1 { ++ //Note: this has been confirmed to malfunction on Intel+NV laptops, ++ // but also on Angle. ++ log::warn!("EGL says it can present to the window but not natively",); ++ } ++ // Android emulator can't natively present either. ++ let tier_threshold = ++ if cfg!(target_os = "android") || cfg!(windows) || cfg!(target_env = "ohos") { ++ 1 ++ } else { ++ 2 ++ }; ++ return Ok((config, tier_max >= tier_threshold)); ++ } ++ Ok(None) => { ++ log::warn!("No config found!"); ++ } ++ Err(e) => { ++ log::error!("error in choose_first_config: {:?}", e); ++ } ++ } ++ } ++ ++ // TODO: include diagnostic details that are currently logged ++ Err(crate::InstanceError::new(String::from( ++ "unable to find an acceptable EGL framebuffer configuration", ++ ))) ++} ++ ++#[derive(Clone, Debug)] ++struct EglContext { ++ instance: Arc, ++ version: (i32, i32), ++ display: khronos_egl::Display, ++ raw: khronos_egl::Context, ++ pbuffer: Option, ++} ++ ++impl EglContext { ++ fn make_current(&self) { ++ self.instance ++ .make_current(self.display, self.pbuffer, self.pbuffer, Some(self.raw)) ++ .unwrap(); ++ } ++ ++ fn unmake_current(&self) { ++ self.instance ++ .make_current(self.display, None, None, None) ++ .unwrap(); ++ } ++} ++ ++/// A wrapper around a [`glow::Context`] and the required EGL context that uses locking to guarantee ++/// exclusive access when shared with multiple threads. ++pub struct AdapterContext { ++ glow: Mutex>, ++ egl: Option, ++} ++ ++unsafe impl Sync for AdapterContext {} ++unsafe impl Send for AdapterContext {} ++ ++impl AdapterContext { ++ pub fn is_owned(&self) -> bool { ++ self.egl.is_some() ++ } ++ ++ /// Returns the EGL instance. ++ /// ++ /// This provides access to EGL functions and the ability to load GL and EGL extension functions. ++ pub fn egl_instance(&self) -> Option<&EglInstance> { ++ self.egl.as_ref().map(|egl| &*egl.instance) ++ } ++ ++ /// Returns the EGLDisplay corresponding to the adapter context. ++ /// ++ /// Returns [`None`] if the adapter was externally created. ++ pub fn raw_display(&self) -> Option<&khronos_egl::Display> { ++ self.egl.as_ref().map(|egl| &egl.display) ++ } ++ ++ /// Returns the EGL version the adapter context was created with. ++ /// ++ /// Returns [`None`] if the adapter was externally created. ++ pub fn egl_version(&self) -> Option<(i32, i32)> { ++ self.egl.as_ref().map(|egl| egl.version) ++ } ++ ++ pub fn raw_context(&self) -> *mut raw::c_void { ++ match self.egl { ++ Some(ref egl) => egl.raw.as_ptr(), ++ None => ptr::null_mut(), ++ } ++ } ++} ++ ++impl Drop for AdapterContext { ++ fn drop(&mut self) { ++ struct CurrentGuard<'a>(&'a EglContext); ++ impl Drop for CurrentGuard<'_> { ++ fn drop(&mut self) { ++ self.0.unmake_current(); ++ } ++ } ++ ++ // Context must be current when dropped. See safety docs on ++ // `glow::HasContext`. ++ // ++ // NOTE: This is only set to `None` by `Adapter::new_external` which ++ // requires the context to be current when anything that may be holding ++ // the `Arc` is dropped. ++ let _guard = self.egl.as_ref().map(|egl| { ++ egl.make_current(); ++ CurrentGuard(egl) ++ }); ++ let glow = self.glow.get_mut(); ++ // SAFETY: Field not used after this. ++ unsafe { ManuallyDrop::drop(glow) }; ++ } ++} ++ ++struct EglContextLock<'a> { ++ instance: &'a Arc, ++ display: khronos_egl::Display, ++} ++ ++/// A guard containing a lock to an [`AdapterContext`], while the GL context is kept current. ++pub struct AdapterContextLock<'a> { ++ glow: MutexGuard<'a, ManuallyDrop>, ++ egl: Option>, ++} ++ ++impl<'a> std::ops::Deref for AdapterContextLock<'a> { ++ type Target = glow::Context; ++ ++ fn deref(&self) -> &Self::Target { ++ &self.glow ++ } ++} ++ ++impl<'a> Drop for AdapterContextLock<'a> { ++ fn drop(&mut self) { ++ if let Some(egl) = self.egl.take() { ++ log::info!("ALVR GLES UNLOCK: releasing wgpu EGL context"); ++ ++ egl.instance ++ .make_current(egl.display, None, None, None) ++ .unwrap(); ++ ++ log::info!("ALVR GLES UNLOCK: wgpu EGL context released"); ++ } ++ } ++} ++ ++impl AdapterContext { ++ /// Get's the [`glow::Context`] without waiting for a lock ++ /// ++ /// # Safety ++ /// ++ /// This should only be called when you have manually made sure that the current thread has made ++ /// the EGL context current and that no other thread also has the EGL context current. ++ /// Additionally, you must manually make the EGL context **not** current after you are done with ++ /// it, so that future calls to `lock()` will not fail. ++ /// ++ /// > **Note:** Calling this function **will** still lock the [`glow::Context`] which adds an ++ /// > extra safe-guard against accidental concurrent access to the context. ++ pub unsafe fn get_without_egl_lock(&self) -> MappedMutexGuard { ++ let guard = self ++ .glow ++ .try_lock_for(Duration::from_secs(CONTEXT_LOCK_TIMEOUT_SECS)) ++ .expect("Could not lock adapter context. This is most-likely a deadlock."); ++ MutexGuard::map(guard, |glow| &mut **glow) ++ } ++ ++ /// Obtain a lock to the EGL context and get handle to the [`glow::Context`] that can be used to ++ /// do rendering. ++ #[track_caller] ++ pub fn lock<'a>(&'a self) -> AdapterContextLock<'a> { ++ let glow = self ++ .glow ++ // Don't lock forever. If it takes longer than 1 second to get the lock we've got a ++ // deadlock and should panic to show where we got stuck ++ .try_lock_for(Duration::from_secs(CONTEXT_LOCK_TIMEOUT_SECS)) ++ .expect("Could not lock adapter context. This is most-likely a deadlock."); ++ ++ let egl = self.egl.as_ref().map(|egl| { ++ log::info!("ALVR GLES LOCK: making wgpu EGL context current"); ++ egl.make_current(); ++ log::info!("ALVR GLES LOCK: wgpu EGL context is current"); ++ EglContextLock { ++ instance: &egl.instance, ++ display: egl.display, ++ } ++ }); ++ ++ AdapterContextLock { glow, egl } ++ } ++} ++ ++#[derive(Debug)] ++struct Inner { ++ /// Note: the context contains a dummy pbuffer (1x1). ++ /// Required for `eglMakeCurrent` on platforms that doesn't supports `EGL_KHR_surfaceless_context`. ++ egl: EglContext, ++ #[allow(unused)] ++ version: (i32, i32), ++ supports_native_window: bool, ++ config: khronos_egl::Config, ++ #[cfg_attr(Emscripten, allow(dead_code))] ++ wl_display: Option<*mut raw::c_void>, ++ #[cfg_attr(Emscripten, allow(dead_code))] ++ force_gles_minor_version: wgt::Gles3MinorVersion, ++ /// Method by which the framebuffer should support srgb ++ srgb_kind: SrgbFrameBufferKind, ++} ++ ++// Different calls to `eglGetPlatformDisplay` may return the same `Display`, making it a global ++// state of all our `EglContext`s. This forces us to track the number of such context to prevent ++// terminating the display if it's currently used by another `EglContext`. ++static DISPLAYS_REFERENCE_COUNT: LazyLock>> = ++ LazyLock::new(Default::default); ++ ++fn initialize_display( ++ egl: &EglInstance, ++ display: khronos_egl::Display, ++) -> Result<(i32, i32), khronos_egl::Error> { ++ let mut guard = DISPLAYS_REFERENCE_COUNT.lock(); ++ *guard.entry(display.as_ptr() as usize).or_default() += 1; ++ ++ // We don't need to check the reference count here since according to the `eglInitialize` ++ // documentation, initializing an already initialized EGL display connection has no effect ++ // besides returning the version numbers. ++ egl.initialize(display) ++} ++ ++fn terminate_display( ++ egl: &EglInstance, ++ display: khronos_egl::Display, ++) -> Result<(), khronos_egl::Error> { ++ let key = &(display.as_ptr() as usize); ++ let mut guard = DISPLAYS_REFERENCE_COUNT.lock(); ++ let count_ref = guard ++ .get_mut(key) ++ .expect("Attempted to decref a display before incref was called"); ++ ++ if *count_ref > 1 { ++ *count_ref -= 1; ++ ++ Ok(()) ++ } else { ++ guard.remove(key); ++ ++ egl.terminate(display) ++ } ++} ++ ++impl Inner { ++ fn create( ++ flags: wgt::InstanceFlags, ++ egl: Arc, ++ display: khronos_egl::Display, ++ force_gles_minor_version: wgt::Gles3MinorVersion, ++ ) -> Result { ++ let version = initialize_display(&egl, display).map_err(|e| { ++ crate::InstanceError::with_source( ++ String::from("failed to initialize EGL display connection"), ++ e, ++ ) ++ })?; ++ let vendor = egl ++ .query_string(Some(display), khronos_egl::VENDOR) ++ .unwrap(); ++ let display_extensions = egl ++ .query_string(Some(display), khronos_egl::EXTENSIONS) ++ .unwrap() ++ .to_string_lossy(); ++ log::debug!("Display vendor {:?}, version {:?}", vendor, version,); ++ log::debug!( ++ "Display extensions: {:#?}", ++ display_extensions.split_whitespace().collect::>() ++ ); ++ ++ let srgb_kind = if version >= (1, 5) { ++ log::debug!("\tEGL surface: +srgb"); ++ SrgbFrameBufferKind::Core ++ } else if display_extensions.contains("EGL_KHR_gl_colorspace") { ++ log::debug!("\tEGL surface: +srgb khr"); ++ SrgbFrameBufferKind::Khr ++ } else { ++ log::warn!("\tEGL surface: -srgb"); ++ SrgbFrameBufferKind::None ++ }; ++ ++ if log::max_level() >= log::LevelFilter::Trace { ++ log::trace!("Configurations:"); ++ let config_count = egl.get_config_count(display).unwrap(); ++ let mut configurations = Vec::with_capacity(config_count); ++ egl.get_configs(display, &mut configurations).unwrap(); ++ for &config in configurations.iter() { ++ log::trace!("\tCONFORMANT=0x{:X}, RENDERABLE=0x{:X}, NATIVE_RENDERABLE=0x{:X}, SURFACE_TYPE=0x{:X}, ALPHA_SIZE={}", ++ egl.get_config_attrib(display, config, khronos_egl::CONFORMANT).unwrap(), ++ egl.get_config_attrib(display, config, khronos_egl::RENDERABLE_TYPE).unwrap(), ++ egl.get_config_attrib(display, config, khronos_egl::NATIVE_RENDERABLE).unwrap(), ++ egl.get_config_attrib(display, config, khronos_egl::SURFACE_TYPE).unwrap(), ++ egl.get_config_attrib(display, config, khronos_egl::ALPHA_SIZE).unwrap(), ++ ); ++ } ++ } ++ ++ let (config, supports_native_window) = choose_config(&egl, display, srgb_kind)?; ++ ++ let supports_opengl = if version >= (1, 4) { ++ let client_apis = egl ++ .query_string(Some(display), khronos_egl::CLIENT_APIS) ++ .unwrap() ++ .to_string_lossy(); ++ client_apis ++ .split(' ') ++ .any(|client_api| client_api == "OpenGL") ++ } else { ++ false ++ }; ++ egl.bind_api(if supports_opengl { ++ khronos_egl::OPENGL_API ++ } else { ++ khronos_egl::OPENGL_ES_API ++ }) ++ .unwrap(); ++ ++ let needs_robustness = true; ++ let mut khr_context_flags = 0; ++ let supports_khr_context = display_extensions.contains("EGL_KHR_create_context"); ++ ++ let mut context_attributes = vec![]; ++ let mut gl_context_attributes = vec![]; ++ let mut gles_context_attributes = vec![]; ++ gl_context_attributes.push(khronos_egl::CONTEXT_MAJOR_VERSION); ++ gl_context_attributes.push(3); ++ gl_context_attributes.push(khronos_egl::CONTEXT_MINOR_VERSION); ++ gl_context_attributes.push(3); ++ if supports_opengl && force_gles_minor_version != wgt::Gles3MinorVersion::Automatic { ++ log::warn!("Ignoring specified GLES minor version as OpenGL is used"); ++ } ++ gles_context_attributes.push(khronos_egl::CONTEXT_MAJOR_VERSION); ++ gles_context_attributes.push(3); // Request GLES 3.0 or higher ++ if force_gles_minor_version != wgt::Gles3MinorVersion::Automatic { ++ gles_context_attributes.push(khronos_egl::CONTEXT_MINOR_VERSION); ++ gles_context_attributes.push(match force_gles_minor_version { ++ wgt::Gles3MinorVersion::Automatic => unreachable!(), ++ wgt::Gles3MinorVersion::Version0 => 0, ++ wgt::Gles3MinorVersion::Version1 => 1, ++ wgt::Gles3MinorVersion::Version2 => 2, ++ }); ++ } ++ if flags.contains(wgt::InstanceFlags::DEBUG) { ++ if version >= (1, 5) { ++ log::debug!("\tEGL context: +debug"); ++ context_attributes.push(khronos_egl::CONTEXT_OPENGL_DEBUG); ++ context_attributes.push(khronos_egl::TRUE as _); ++ } else if supports_khr_context { ++ log::debug!("\tEGL context: +debug KHR"); ++ khr_context_flags |= EGL_CONTEXT_OPENGL_DEBUG_BIT_KHR; ++ } else { ++ log::debug!("\tEGL context: -debug"); ++ } ++ } ++ if needs_robustness { ++ //Note: the core version can fail if robustness is not supported ++ // (regardless of whether the extension is supported!). ++ // In fact, Angle does precisely that awful behavior, so we don't try it there. ++ if version >= (1, 5) && !display_extensions.contains("EGL_ANGLE_") { ++ log::debug!("\tEGL context: +robust access"); ++ context_attributes.push(khronos_egl::CONTEXT_OPENGL_ROBUST_ACCESS); ++ context_attributes.push(khronos_egl::TRUE as _); ++ } else if display_extensions.contains("EGL_EXT_create_context_robustness") { ++ log::debug!("\tEGL context: +robust access EXT"); ++ context_attributes.push(EGL_CONTEXT_OPENGL_ROBUST_ACCESS_EXT); ++ context_attributes.push(khronos_egl::TRUE as _); ++ } else { ++ //Note: we aren't trying `EGL_CONTEXT_OPENGL_ROBUST_ACCESS_BIT_KHR` ++ // because it's for desktop GL only, not GLES. ++ log::warn!("\tEGL context: -robust access"); ++ } ++ } ++ if khr_context_flags != 0 { ++ context_attributes.push(EGL_CONTEXT_FLAGS_KHR); ++ context_attributes.push(khr_context_flags); ++ } ++ context_attributes.push(khronos_egl::NONE); ++ ++ gl_context_attributes.extend(&context_attributes); ++ gles_context_attributes.extend(&context_attributes); ++ ++ // ALVR: share GL objects with the app's current EGL context, so textures created by the ++ // app (glGenTextures) are visible inside this context. ++ let share_context = egl.get_current_context(); ++ log::info!("ALVR GLES: share context = {:?}", share_context.map(|c| c.as_ptr())); ++ ++ let context = if supports_opengl { ++ egl.create_context(display, config, None, &gl_context_attributes) ++ .or_else(|_| { ++ egl.bind_api(khronos_egl::OPENGL_ES_API).unwrap(); ++ egl.create_context(display, config, None, &gles_context_attributes) ++ }) ++ .map_err(|e| { ++ crate::InstanceError::with_source( ++ String::from("unable to create OpenGL or GLES 3.x context"), ++ e, ++ ) ++ }) ++ } else { ++ egl.create_context(display, config, None, &gles_context_attributes) ++ .map_err(|e| { ++ crate::InstanceError::with_source( ++ String::from("unable to create GLES 3.x context"), ++ e, ++ ) ++ }) ++ }?; ++ ++ // Testing if context can be binded without surface ++ // and creating dummy pbuffer surface if not. ++ let pbuffer = if version >= (1, 5) ++ || display_extensions.contains("EGL_KHR_surfaceless_context") ++ || cfg!(Emscripten) ++ { ++ log::debug!("\tEGL context: +surfaceless"); ++ None ++ } else { ++ let attributes = [ ++ khronos_egl::WIDTH, ++ 1, ++ khronos_egl::HEIGHT, ++ 1, ++ khronos_egl::NONE, ++ ]; ++ egl.create_pbuffer_surface(display, config, &attributes) ++ .map(Some) ++ .map_err(|e| { ++ crate::InstanceError::with_source( ++ String::from("error in create_pbuffer_surface"), ++ e, ++ ) ++ })? ++ }; ++ ++ Ok(Self { ++ egl: EglContext { ++ instance: egl, ++ display, ++ raw: context, ++ pbuffer, ++ version, ++ }, ++ version, ++ supports_native_window, ++ config, ++ wl_display: None, ++ srgb_kind, ++ force_gles_minor_version, ++ }) ++ } ++} ++ ++impl Drop for Inner { ++ fn drop(&mut self) { ++ if let Err(e) = self ++ .egl ++ .instance ++ .destroy_context(self.egl.display, self.egl.raw) ++ { ++ log::warn!("Error in destroy_context: {:?}", e); ++ } ++ ++ if let Err(e) = terminate_display(&self.egl.instance, self.egl.display) { ++ log::warn!("Error in terminate: {:?}", e); ++ } ++ } ++} ++ ++#[derive(Clone, Copy, Debug, PartialEq)] ++enum WindowKind { ++ Wayland, ++ X11, ++ AngleX11, ++ Unknown, ++} ++ ++#[derive(Clone, Debug)] ++struct WindowSystemInterface { ++ display_owner: Option>, ++ kind: WindowKind, ++} ++ ++pub struct Instance { ++ wsi: WindowSystemInterface, ++ flags: wgt::InstanceFlags, ++ options: wgt::GlBackendOptions, ++ inner: Mutex, ++} ++ ++impl Instance { ++ pub fn raw_display(&self) -> khronos_egl::Display { ++ self.inner ++ .try_lock() ++ .expect("Could not lock instance. This is most-likely a deadlock.") ++ .egl ++ .display ++ } ++ ++ /// Returns the version of the EGL display. ++ pub fn egl_version(&self) -> (i32, i32) { ++ self.inner ++ .try_lock() ++ .expect("Could not lock instance. This is most-likely a deadlock.") ++ .version ++ } ++ ++ pub fn egl_config(&self) -> khronos_egl::Config { ++ self.inner ++ .try_lock() ++ .expect("Could not lock instance. This is most-likely a deadlock.") ++ .config ++ } ++} ++ ++unsafe impl Send for Instance {} ++unsafe impl Sync for Instance {} ++ ++impl crate::Instance for Instance { ++ type A = super::Api; ++ ++ unsafe fn init(desc: &crate::InstanceDescriptor) -> Result { ++ profiling::scope!("Init OpenGL (EGL) Backend"); ++ #[cfg(Emscripten)] ++ let egl_result: Result = ++ Ok(khronos_egl::Instance::new(khronos_egl::Static)); ++ ++ #[cfg(not(Emscripten))] ++ let egl_result = if cfg!(windows) { ++ unsafe { ++ khronos_egl::DynamicInstance::::load_required_from_filename( ++ "libEGL.dll", ++ ) ++ } ++ } else if cfg!(target_vendor = "apple") { ++ unsafe { ++ khronos_egl::DynamicInstance::::load_required_from_filename( ++ "libEGL.dylib", ++ ) ++ } ++ } else { ++ unsafe { khronos_egl::DynamicInstance::::load_required() } ++ }; ++ let egl = match egl_result { ++ Ok(egl) => Arc::new(egl), ++ Err(e) => { ++ return Err(crate::InstanceError::with_source( ++ String::from("unable to open libEGL"), ++ e, ++ )); ++ } ++ }; ++ ++ let client_extensions = egl.query_string(None, khronos_egl::EXTENSIONS); ++ ++ let client_ext_str = match client_extensions { ++ Ok(ext) => ext.to_string_lossy().into_owned(), ++ Err(_) => String::new(), ++ }; ++ log::debug!( ++ "Client extensions: {:#?}", ++ client_ext_str.split_whitespace().collect::>() ++ ); ++ ++ let wayland_library = if client_ext_str.contains("EGL_EXT_platform_wayland") { ++ test_wayland_display() ++ } else { ++ None ++ }; ++ let x11_display_library = if client_ext_str.contains("EGL_EXT_platform_x11") { ++ open_x_display() ++ } else { ++ None ++ }; ++ let angle_x11_display_library = if client_ext_str.contains("EGL_ANGLE_platform_angle") { ++ open_x_display() ++ } else { ++ None ++ }; ++ ++ #[cfg(not(Emscripten))] ++ let egl1_5 = egl.upcast::(); ++ ++ #[cfg(Emscripten)] ++ let egl1_5: Option<&Arc> = Some(&egl); ++ ++ let (display, display_owner, wsi_kind) = ++ if let (Some(library), Some(egl)) = (wayland_library, egl1_5) { ++ log::info!("Using Wayland platform"); ++ let display_attributes = [khronos_egl::ATTRIB_NONE]; ++ let display = unsafe { ++ egl.get_platform_display( ++ EGL_PLATFORM_WAYLAND_KHR, ++ khronos_egl::DEFAULT_DISPLAY, ++ &display_attributes, ++ ) ++ } ++ .unwrap(); ++ (display, Some(Rc::new(library)), WindowKind::Wayland) ++ } else if let (Some(display_owner), Some(egl)) = (x11_display_library, egl1_5) { ++ log::info!("Using X11 platform"); ++ let display_attributes = [khronos_egl::ATTRIB_NONE]; ++ let display = unsafe { ++ egl.get_platform_display( ++ EGL_PLATFORM_X11_KHR, ++ display_owner.display.as_ptr(), ++ &display_attributes, ++ ) ++ } ++ .unwrap(); ++ (display, Some(Rc::new(display_owner)), WindowKind::X11) ++ } else if let (Some(display_owner), Some(egl)) = (angle_x11_display_library, egl1_5) { ++ log::info!("Using Angle platform with X11"); ++ let display_attributes = [ ++ EGL_PLATFORM_ANGLE_NATIVE_PLATFORM_TYPE_ANGLE as khronos_egl::Attrib, ++ EGL_PLATFORM_X11_KHR as khronos_egl::Attrib, ++ EGL_PLATFORM_ANGLE_DEBUG_LAYERS_ENABLED as khronos_egl::Attrib, ++ usize::from(desc.flags.contains(wgt::InstanceFlags::VALIDATION)), ++ khronos_egl::ATTRIB_NONE, ++ ]; ++ let display = unsafe { ++ egl.get_platform_display( ++ EGL_PLATFORM_ANGLE_ANGLE, ++ display_owner.display.as_ptr(), ++ &display_attributes, ++ ) ++ } ++ .unwrap(); ++ (display, Some(Rc::new(display_owner)), WindowKind::AngleX11) ++ } else if client_ext_str.contains("EGL_MESA_platform_surfaceless") { ++ log::warn!("No windowing system present. Using surfaceless platform"); ++ #[allow(clippy::unnecessary_literal_unwrap)] // This is only a literal on Emscripten ++ let egl = egl1_5.expect("Failed to get EGL 1.5 for surfaceless"); ++ let display = unsafe { ++ egl.get_platform_display( ++ EGL_PLATFORM_SURFACELESS_MESA, ++ khronos_egl::DEFAULT_DISPLAY, ++ &[khronos_egl::ATTRIB_NONE], ++ ) ++ } ++ .unwrap(); ++ ++ (display, None, WindowKind::Unknown) ++ } else { ++ log::warn!("EGL_MESA_platform_surfaceless not available. Using default platform"); ++ let display = unsafe { egl.get_display(khronos_egl::DEFAULT_DISPLAY) }.unwrap(); ++ (display, None, WindowKind::Unknown) ++ }; ++ ++ if desc.flags.contains(wgt::InstanceFlags::VALIDATION) ++ && client_ext_str.contains("EGL_KHR_debug") ++ { ++ log::debug!("Enabling EGL debug output"); ++ let function: EglDebugMessageControlFun = { ++ let addr = egl.get_proc_address("eglDebugMessageControlKHR").unwrap(); ++ unsafe { std::mem::transmute(addr) } ++ }; ++ let attributes = [ ++ EGL_DEBUG_MSG_CRITICAL_KHR as khronos_egl::Attrib, ++ 1, ++ EGL_DEBUG_MSG_ERROR_KHR as khronos_egl::Attrib, ++ 1, ++ EGL_DEBUG_MSG_WARN_KHR as khronos_egl::Attrib, ++ 1, ++ EGL_DEBUG_MSG_INFO_KHR as khronos_egl::Attrib, ++ 1, ++ khronos_egl::ATTRIB_NONE, ++ ]; ++ unsafe { (function)(Some(egl_debug_proc), attributes.as_ptr()) }; ++ } ++ ++ let inner = Inner::create( ++ desc.flags, ++ egl, ++ display, ++ desc.backend_options.gl.gles_minor_version, ++ )?; ++ ++ Ok(Instance { ++ wsi: WindowSystemInterface { ++ display_owner, ++ kind: wsi_kind, ++ }, ++ flags: desc.flags, ++ options: desc.backend_options.gl.clone(), ++ inner: Mutex::new(inner), ++ }) ++ } ++ ++ #[cfg_attr(target_os = "macos", allow(unused, unused_mut, unreachable_code))] ++ unsafe fn create_surface( ++ &self, ++ display_handle: raw_window_handle::RawDisplayHandle, ++ window_handle: raw_window_handle::RawWindowHandle, ++ ) -> Result { ++ use raw_window_handle::RawWindowHandle as Rwh; ++ ++ #[cfg_attr(any(target_os = "android", Emscripten), allow(unused_mut))] ++ let mut inner = self.inner.lock(); ++ ++ match (window_handle, display_handle) { ++ (Rwh::Xlib(_), _) => {} ++ (Rwh::Xcb(_), _) => {} ++ (Rwh::Win32(_), _) => {} ++ (Rwh::AppKit(_), _) => {} ++ (Rwh::OhosNdk(_), _) => {} ++ #[cfg(target_os = "android")] ++ (Rwh::AndroidNdk(handle), _) => { ++ let format = inner ++ .egl ++ .instance ++ .get_config_attrib( ++ inner.egl.display, ++ inner.config, ++ khronos_egl::NATIVE_VISUAL_ID, ++ ) ++ .unwrap(); ++ ++ let ret = unsafe { ++ ndk_sys::ANativeWindow_setBuffersGeometry( ++ handle ++ .a_native_window ++ .as_ptr() ++ .cast::(), ++ 0, ++ 0, ++ format, ++ ) ++ }; ++ ++ if ret != 0 { ++ return Err(crate::InstanceError::new(format!( ++ "error {ret} returned from ANativeWindow_setBuffersGeometry", ++ ))); ++ } ++ } ++ #[cfg(not(Emscripten))] ++ (Rwh::Wayland(_), raw_window_handle::RawDisplayHandle::Wayland(display_handle)) => { ++ if inner ++ .wl_display ++ .map(|ptr| ptr != display_handle.display.as_ptr()) ++ .unwrap_or(true) ++ { ++ /* Wayland displays are not sharable between surfaces so if the ++ * surface we receive from this handle is from a different ++ * display, we must re-initialize the context. ++ * ++ * See gfx-rs/gfx#3545 ++ */ ++ log::warn!("Re-initializing Gles context due to Wayland window"); ++ ++ use std::ops::DerefMut; ++ let display_attributes = [khronos_egl::ATTRIB_NONE]; ++ ++ let display = unsafe { ++ inner ++ .egl ++ .instance ++ .upcast::() ++ .unwrap() ++ .get_platform_display( ++ EGL_PLATFORM_WAYLAND_KHR, ++ display_handle.display.as_ptr(), ++ &display_attributes, ++ ) ++ } ++ .unwrap(); ++ ++ let new_inner = Inner::create( ++ self.flags, ++ Arc::clone(&inner.egl.instance), ++ display, ++ inner.force_gles_minor_version, ++ )?; ++ ++ let old_inner = std::mem::replace(inner.deref_mut(), new_inner); ++ inner.wl_display = Some(display_handle.display.as_ptr()); ++ ++ drop(old_inner); ++ } ++ } ++ #[cfg(Emscripten)] ++ (Rwh::Web(_), _) => {} ++ other => { ++ return Err(crate::InstanceError::new(format!( ++ "unsupported window: {other:?}" ++ ))); ++ } ++ }; ++ ++ inner.egl.unmake_current(); ++ ++ Ok(Surface { ++ egl: inner.egl.clone(), ++ wsi: self.wsi.clone(), ++ config: inner.config, ++ presentable: inner.supports_native_window, ++ raw_window_handle: window_handle, ++ swapchain: RwLock::new(None), ++ srgb_kind: inner.srgb_kind, ++ }) ++ } ++ ++ unsafe fn enumerate_adapters( ++ &self, ++ _surface_hint: Option<&Surface>, ++ ) -> Vec> { ++ let inner = self.inner.lock(); ++ inner.egl.make_current(); ++ ++ let mut gl = unsafe { ++ glow::Context::from_loader_function(|name| { ++ inner ++ .egl ++ .instance ++ .get_proc_address(name) ++ .map_or(ptr::null(), |p| p as *const _) ++ }) ++ }; ++ ++ // In contrast to OpenGL ES, OpenGL requires explicitly enabling sRGB conversions, ++ // as otherwise the user has to do the sRGB conversion. ++ if !matches!(inner.srgb_kind, SrgbFrameBufferKind::None) { ++ unsafe { gl.enable(glow::FRAMEBUFFER_SRGB) }; ++ } ++ ++ if self.flags.contains(wgt::InstanceFlags::DEBUG) && gl.supports_debug() { ++ log::debug!("Max label length: {}", unsafe { ++ gl.get_parameter_i32(glow::MAX_LABEL_LENGTH) ++ }); ++ } ++ ++ if self.flags.contains(wgt::InstanceFlags::VALIDATION) && gl.supports_debug() { ++ log::debug!("Enabling GLES debug output"); ++ unsafe { gl.enable(glow::DEBUG_OUTPUT) }; ++ unsafe { gl.debug_message_callback(super::gl_debug_message_callback) }; ++ } ++ ++ // Wrap in ManuallyDrop to make it easier to "current" the GL context before dropping this ++ // GLOW context, which could also happen if a panic occurs after we uncurrent the context ++ // below but before AdapterContext is constructed. ++ let gl = ManuallyDrop::new(gl); ++ inner.egl.unmake_current(); ++ ++ unsafe { ++ super::Adapter::expose( ++ AdapterContext { ++ glow: Mutex::new(gl), ++ egl: Some(inner.egl.clone()), ++ }, ++ self.options.clone(), ++ ) ++ } ++ .into_iter() ++ .collect() ++ } ++} ++ ++impl super::Adapter { ++ /// Creates a new external adapter using the specified loader function. ++ /// ++ /// # Safety ++ /// ++ /// - The underlying OpenGL ES context must be current. ++ /// - The underlying OpenGL ES context must be current when interfacing with any objects returned by ++ /// wgpu-hal from this adapter. ++ /// - The underlying OpenGL ES context must be current when dropping this adapter and when ++ /// dropping any objects returned from this adapter. ++ pub unsafe fn new_external( ++ fun: impl FnMut(&str) -> *const ffi::c_void, ++ options: wgt::GlBackendOptions, ++ ) -> Option> { ++ let context = unsafe { glow::Context::from_loader_function(fun) }; ++ unsafe { ++ Self::expose( ++ AdapterContext { ++ glow: Mutex::new(ManuallyDrop::new(context)), ++ egl: None, ++ }, ++ options, ++ ) ++ } ++ } ++ ++ pub fn adapter_context(&self) -> &AdapterContext { ++ &self.shared.context ++ } ++} ++ ++impl super::Device { ++ /// Returns the underlying EGL context. ++ pub fn context(&self) -> &AdapterContext { ++ &self.shared.context ++ } ++} ++ ++#[derive(Debug)] ++pub struct Swapchain { ++ surface: khronos_egl::Surface, ++ wl_window: Option<*mut raw::c_void>, ++ framebuffer: glow::Framebuffer, ++ renderbuffer: glow::Renderbuffer, ++ /// Extent because the window lies ++ extent: wgt::Extent3d, ++ format: wgt::TextureFormat, ++ format_desc: super::TextureFormatDesc, ++ #[allow(unused)] ++ sample_type: wgt::TextureSampleType, ++} ++ ++#[derive(Debug)] ++pub struct Surface { ++ egl: EglContext, ++ wsi: WindowSystemInterface, ++ config: khronos_egl::Config, ++ pub(super) presentable: bool, ++ raw_window_handle: raw_window_handle::RawWindowHandle, ++ swapchain: RwLock>, ++ srgb_kind: SrgbFrameBufferKind, ++} ++ ++unsafe impl Send for Surface {} ++unsafe impl Sync for Surface {} ++ ++impl Surface { ++ pub(super) unsafe fn present( ++ &self, ++ _suf_texture: super::Texture, ++ context: &AdapterContext, ++ ) -> Result<(), crate::SurfaceError> { ++ let gl = unsafe { context.get_without_egl_lock() }; ++ let swapchain = self.swapchain.read(); ++ let sc = swapchain.as_ref().unwrap(); ++ ++ self.egl ++ .instance ++ .make_current( ++ self.egl.display, ++ Some(sc.surface), ++ Some(sc.surface), ++ Some(self.egl.raw), ++ ) ++ .map_err(|e| { ++ log::error!("make_current(surface) failed: {}", e); ++ crate::SurfaceError::Lost ++ })?; ++ ++ unsafe { gl.disable(glow::SCISSOR_TEST) }; ++ unsafe { gl.color_mask(true, true, true, true) }; ++ ++ unsafe { gl.bind_framebuffer(glow::DRAW_FRAMEBUFFER, None) }; ++ unsafe { gl.bind_framebuffer(glow::READ_FRAMEBUFFER, Some(sc.framebuffer)) }; ++ ++ if !matches!(self.srgb_kind, SrgbFrameBufferKind::None) { ++ // Disable sRGB conversions for `glBlitFramebuffer` as behavior does diverge between ++ // drivers and formats otherwise and we want to ensure no sRGB conversions happen. ++ unsafe { gl.disable(glow::FRAMEBUFFER_SRGB) }; ++ } ++ ++ // Note the Y-flipping here. GL's presentation is not flipped, ++ // but main rendering is. Therefore, we Y-flip the output positions ++ // in the shader, and also this blit. ++ unsafe { ++ gl.blit_framebuffer( ++ 0, ++ sc.extent.height as i32, ++ sc.extent.width as i32, ++ 0, ++ 0, ++ 0, ++ sc.extent.width as i32, ++ sc.extent.height as i32, ++ glow::COLOR_BUFFER_BIT, ++ glow::NEAREST, ++ ) ++ }; ++ ++ if !matches!(self.srgb_kind, SrgbFrameBufferKind::None) { ++ unsafe { gl.enable(glow::FRAMEBUFFER_SRGB) }; ++ } ++ ++ unsafe { gl.bind_framebuffer(glow::READ_FRAMEBUFFER, None) }; ++ ++ self.egl ++ .instance ++ .swap_buffers(self.egl.display, sc.surface) ++ .map_err(|e| { ++ log::error!("swap_buffers failed: {}", e); ++ crate::SurfaceError::Lost ++ // TODO: should we unset the current context here? ++ })?; ++ self.egl ++ .instance ++ .make_current(self.egl.display, None, None, None) ++ .map_err(|e| { ++ log::error!("make_current(null) failed: {}", e); ++ crate::SurfaceError::Lost ++ })?; ++ ++ Ok(()) ++ } ++ ++ unsafe fn unconfigure_impl( ++ &self, ++ device: &super::Device, ++ ) -> Option<(khronos_egl::Surface, Option<*mut raw::c_void>)> { ++ let gl = &device.shared.context.lock(); ++ match self.swapchain.write().take() { ++ Some(sc) => { ++ unsafe { gl.delete_renderbuffer(sc.renderbuffer) }; ++ unsafe { gl.delete_framebuffer(sc.framebuffer) }; ++ Some((sc.surface, sc.wl_window)) ++ } ++ None => None, ++ } ++ } ++ ++ pub fn supports_srgb(&self) -> bool { ++ match self.srgb_kind { ++ SrgbFrameBufferKind::None => false, ++ _ => true, ++ } ++ } ++} ++ ++impl crate::Surface for Surface { ++ type A = super::Api; ++ ++ unsafe fn configure( ++ &self, ++ device: &super::Device, ++ config: &crate::SurfaceConfiguration, ++ ) -> Result<(), crate::SurfaceError> { ++ use raw_window_handle::RawWindowHandle as Rwh; ++ ++ let (surface, wl_window) = match unsafe { self.unconfigure_impl(device) } { ++ Some(pair) => pair, ++ None => { ++ let mut wl_window = None; ++ let (mut temp_xlib_handle, mut temp_xcb_handle); ++ let native_window_ptr = match (self.wsi.kind, self.raw_window_handle) { ++ (WindowKind::Unknown | WindowKind::X11, Rwh::Xlib(handle)) => { ++ temp_xlib_handle = handle.window; ++ ptr::from_mut(&mut temp_xlib_handle).cast::() ++ } ++ (WindowKind::AngleX11, Rwh::Xlib(handle)) => handle.window as *mut ffi::c_void, ++ (WindowKind::Unknown | WindowKind::X11, Rwh::Xcb(handle)) => { ++ temp_xcb_handle = handle.window; ++ ptr::from_mut(&mut temp_xcb_handle).cast::() ++ } ++ (WindowKind::AngleX11, Rwh::Xcb(handle)) => { ++ handle.window.get() as *mut ffi::c_void ++ } ++ (WindowKind::Unknown, Rwh::AndroidNdk(handle)) => { ++ handle.a_native_window.as_ptr() ++ } ++ (WindowKind::Unknown, Rwh::OhosNdk(handle)) => handle.native_window.as_ptr(), ++ (WindowKind::Wayland, Rwh::Wayland(handle)) => { ++ let library = &self.wsi.display_owner.as_ref().unwrap().library; ++ let wl_egl_window_create: libloading::Symbol = ++ unsafe { library.get(c"wl_egl_window_create".to_bytes()) }.unwrap(); ++ let window = ++ unsafe { wl_egl_window_create(handle.surface.as_ptr(), 640, 480) } ++ .cast(); ++ wl_window = Some(window); ++ window ++ } ++ #[cfg(Emscripten)] ++ (WindowKind::Unknown, Rwh::Web(handle)) => handle.id as *mut ffi::c_void, ++ (WindowKind::Unknown, Rwh::Win32(handle)) => { ++ handle.hwnd.get() as *mut ffi::c_void ++ } ++ (WindowKind::Unknown, Rwh::AppKit(handle)) => { ++ #[cfg(not(target_os = "macos"))] ++ let window_ptr = handle.ns_view.as_ptr(); ++ #[cfg(target_os = "macos")] ++ let window_ptr = { ++ use objc::{msg_send, runtime::Object, sel, sel_impl}; ++ // ns_view always have a layer and don't need to verify that it exists. ++ let layer: *mut Object = ++ msg_send![handle.ns_view.as_ptr().cast::(), layer]; ++ layer.cast::() ++ }; ++ window_ptr ++ } ++ _ => { ++ log::warn!( ++ "Initialized platform {:?} doesn't work with window {:?}", ++ self.wsi.kind, ++ self.raw_window_handle ++ ); ++ return Err(crate::SurfaceError::Other("incompatible window kind")); ++ } ++ }; ++ ++ let mut attributes = vec![ ++ khronos_egl::RENDER_BUFFER, ++ // We don't want any of the buffering done by the driver, because we ++ // manage a swapchain on our side. ++ // Some drivers just fail on surface creation seeing `EGL_SINGLE_BUFFER`. ++ if cfg!(any( ++ target_os = "android", ++ target_os = "macos", ++ target_env = "ohos" ++ )) || cfg!(windows) ++ || self.wsi.kind == WindowKind::AngleX11 ++ { ++ khronos_egl::BACK_BUFFER ++ } else { ++ khronos_egl::SINGLE_BUFFER ++ }, ++ ]; ++ if config.format.is_srgb() { ++ match self.srgb_kind { ++ SrgbFrameBufferKind::None => {} ++ SrgbFrameBufferKind::Core => { ++ attributes.push(khronos_egl::GL_COLORSPACE); ++ attributes.push(khronos_egl::GL_COLORSPACE_SRGB); ++ } ++ SrgbFrameBufferKind::Khr => { ++ attributes.push(EGL_GL_COLORSPACE_KHR as i32); ++ attributes.push(EGL_GL_COLORSPACE_SRGB_KHR as i32); ++ } ++ } ++ } ++ attributes.push(khronos_egl::ATTRIB_NONE as i32); ++ ++ #[cfg(not(Emscripten))] ++ let egl1_5 = self.egl.instance.upcast::(); ++ ++ #[cfg(Emscripten)] ++ let egl1_5: Option<&Arc> = Some(&self.egl.instance); ++ ++ // Careful, we can still be in 1.4 version even if `upcast` succeeds ++ let raw_result = match egl1_5 { ++ Some(egl) if self.wsi.kind != WindowKind::Unknown => { ++ let attributes_usize = attributes ++ .into_iter() ++ .map(|v| v as usize) ++ .collect::>(); ++ unsafe { ++ egl.create_platform_window_surface( ++ self.egl.display, ++ self.config, ++ native_window_ptr, ++ &attributes_usize, ++ ) ++ } ++ } ++ _ => unsafe { ++ self.egl.instance.create_window_surface( ++ self.egl.display, ++ self.config, ++ native_window_ptr, ++ Some(&attributes), ++ ) ++ }, ++ }; ++ ++ match raw_result { ++ Ok(raw) => (raw, wl_window), ++ Err(e) => { ++ log::warn!("Error in create_window_surface: {:?}", e); ++ return Err(crate::SurfaceError::Lost); ++ } ++ } ++ } ++ }; ++ ++ if let Some(window) = wl_window { ++ let library = &self.wsi.display_owner.as_ref().unwrap().library; ++ let wl_egl_window_resize: libloading::Symbol = ++ unsafe { library.get(c"wl_egl_window_resize".to_bytes()) }.unwrap(); ++ unsafe { ++ wl_egl_window_resize( ++ window, ++ config.extent.width as i32, ++ config.extent.height as i32, ++ 0, ++ 0, ++ ) ++ }; ++ } ++ ++ let format_desc = device.shared.describe_texture_format(config.format); ++ let gl = &device.shared.context.lock(); ++ let renderbuffer = unsafe { gl.create_renderbuffer() }.map_err(|error| { ++ log::error!("Internal swapchain renderbuffer creation failed: {error}"); ++ crate::DeviceError::OutOfMemory ++ })?; ++ unsafe { gl.bind_renderbuffer(glow::RENDERBUFFER, Some(renderbuffer)) }; ++ unsafe { ++ gl.renderbuffer_storage( ++ glow::RENDERBUFFER, ++ format_desc.internal, ++ config.extent.width as _, ++ config.extent.height as _, ++ ) ++ }; ++ let framebuffer = unsafe { gl.create_framebuffer() }.map_err(|error| { ++ log::error!("Internal swapchain framebuffer creation failed: {error}"); ++ crate::DeviceError::OutOfMemory ++ })?; ++ unsafe { gl.bind_framebuffer(glow::READ_FRAMEBUFFER, Some(framebuffer)) }; ++ unsafe { ++ gl.framebuffer_renderbuffer( ++ glow::READ_FRAMEBUFFER, ++ glow::COLOR_ATTACHMENT0, ++ glow::RENDERBUFFER, ++ Some(renderbuffer), ++ ) ++ }; ++ unsafe { gl.bind_renderbuffer(glow::RENDERBUFFER, None) }; ++ unsafe { gl.bind_framebuffer(glow::READ_FRAMEBUFFER, None) }; ++ ++ let mut swapchain = self.swapchain.write(); ++ *swapchain = Some(Swapchain { ++ surface, ++ wl_window, ++ renderbuffer, ++ framebuffer, ++ extent: config.extent, ++ format: config.format, ++ format_desc, ++ sample_type: wgt::TextureSampleType::Float { filterable: false }, ++ }); ++ ++ Ok(()) ++ } ++ ++ unsafe fn unconfigure(&self, device: &super::Device) { ++ if let Some((surface, wl_window)) = unsafe { self.unconfigure_impl(device) } { ++ self.egl ++ .instance ++ .destroy_surface(self.egl.display, surface) ++ .unwrap(); ++ if let Some(window) = wl_window { ++ let library = &self ++ .wsi ++ .display_owner ++ .as_ref() ++ .expect("unsupported window") ++ .library; ++ let wl_egl_window_destroy: libloading::Symbol = ++ unsafe { library.get(c"wl_egl_window_destroy".to_bytes()) }.unwrap(); ++ unsafe { wl_egl_window_destroy(window) }; ++ } ++ } ++ } ++ ++ unsafe fn acquire_texture( ++ &self, ++ _timeout_ms: Option, //TODO ++ _fence: &super::Fence, ++ ) -> Result>, crate::SurfaceError> { ++ let swapchain = self.swapchain.read(); ++ let sc = swapchain.as_ref().unwrap(); ++ let texture = super::Texture { ++ inner: super::TextureInner::Renderbuffer { ++ raw: sc.renderbuffer, ++ }, ++ drop_guard: None, ++ array_layer_count: 1, ++ mip_level_count: 1, ++ format: sc.format, ++ format_desc: sc.format_desc.clone(), ++ copy_size: crate::CopyExtent { ++ width: sc.extent.width, ++ height: sc.extent.height, ++ depth: 1, ++ }, ++ }; ++ Ok(Some(crate::AcquiredSurfaceTexture { ++ texture, ++ suboptimal: false, ++ })) ++ } ++ unsafe fn discard_texture(&self, _texture: super::Texture) {} ++} +diff -ruN a/src/gles/queue.rs b/src/gles/queue.rs +--- a/src/gles/queue.rs 2026-10-10 19:59:35.659717497 +0900 ++++ b/src/gles/queue.rs 2026-10-10 19:59:35.673313980 +0900 @@ -144,7 +144,14 @@ get_2d_target(target, view.array_layers.start), Some(raw), From 8db689e60b41f5fceed3a6b37b0f7ef9c14fd762 Mon Sep 17 00:00:00 2001 From: Rob Oates Date: Sat, 10 Oct 2026 20:25:54 +0900 Subject: [PATCH 17/29] Update migration notes: ARCore timing, PowerVR debug-output crash, tested devices --- notes/MIGRATION_NOTES.md | 84 ++++++++++++++++++++++++++++++++-------- 1 file changed, 68 insertions(+), 16 deletions(-) diff --git a/notes/MIGRATION_NOTES.md b/notes/MIGRATION_NOTES.md index 36383d51..8b67e57a 100644 --- a/notes/MIGRATION_NOTES.md +++ b/notes/MIGRATION_NOTES.md @@ -1,27 +1,34 @@ # PhoneVR: ALVR v20 -> v21 migration notes -Last updated: 2026-10-05 +Last updated: 2026-10-10 ## Status -The Android client (the `main` flavor, built on `alvr_client_core`) now connects to an ALVR v21 +The Android client (the `main` flavor, built on `alvr_client_core`) connects to an ALVR v21 streamer, decodes video on the phone's hardware decoder (H.264 and HEVC verified) and renders it through the Cardboard distortion renderer. Head orientation comes from the Cardboard head tracker -(3DoF). Position is not tracked yet. +and, on phones with ARCore, ARCore adds position tracking and corrects Cardboard's yaw drift +(see `ARCORE_NOTES.md`). On phones without ARCore the client runs as 3DoF. - **ALVR version:** v21 is not a tagged release. It is the `master` branch of `alvr-org/ALVR` (the main repo's newest tag is still v20.14.1). This port was built against commit `9f118394` (`21.0.0-dev14`). The client library header from `dc84d683` (also `21.0.0-dev14`) is identical. -- **Test device:** an Android phone with an Adreno 619 GPU, OpenGL ES 3.2, with - `GL_OES_EGL_image_external` supported. - **Build host:** Ubuntu in VirtualBox, JDK 17, Rust 1.97.1, Android NDK 25.2.9519653. +### Tested devices + +| Phone | Android | GPU | ARCore | Result | +|---|---|---|---|---| +| Oppo Reno 9 A | 14 | Adreno 619 | yes | Works: video, 6DoF position, drift-free yaw | +| AQUOS sense4 basic | 12 | Adreno 618 | yes | Works after section 4.4 (start ARCore after video) | +| UMIDIGI A3S | 10 | PowerVR Rogue GE8300 (MediaTek) | no | Works as 3DoF after section 4.5 (GL debug output off) | + ## 1. Build configuration changes -No source folders were removed in this branch. `app/src/gvr/`, `app/src/androidTestGvr/` and -`code/mobile/mobile-common/` are still present. They belong to the legacy custom-protocol client -and should no longer be built, because the build files below were changed to produce only the -ALVR-based variant. They can be deleted in a later cleanup. +> **TODO (verify before publishing):** the first version of these notes said the folders +> `app/src/gvr/`, `app/src/androidTestGvr/` and `code/mobile/mobile-common/` were deleted. Check +> with `ls` whether they still exist in this branch. If they do, delete this note and the +> assumption that they are gone; if they were removed, say so here. The build files were changed to produce a single ALVR-based variant (from the original port notes): @@ -72,7 +79,8 @@ differences. The Gradle `buildClientLib` task regenerates this file on every bui `alvr_initialize_opengl()` while its own context is current (it does, in `surfaceCreatedNative`). The patch is in `code/mobile/android/PhoneVR/patches/wgpu-hal-25.0.2-share-egl-context.patch`. It also contains some `ALVR GLES` debug log lines that can be dropped; the essential change is - replacing `None` with `share_context` in the `create_context` calls. + replacing `None` with `share_context` in the `create_context` calls. The same patch also holds the + change from section 4.5 (GL debug output and the debug EGL context switched off). - **Related workaround:** the `alvr_*_opengl()` calls make Rust's context current and leave the thread with none, so `alvr_main.cpp` restores the app's EGL state afterwards (`EglStateGuard`). @@ -97,6 +105,41 @@ differences. The Gradle `buildClientLib` task regenerates this file on every bui H.264 and HEVC. The generic `low-latency` key is deliberately not set; ALVR's own settings code notes it crashes some Android phones. +### 4.4 ARCore's camera stops decoded video on some phones + +- **Symptom:** on an AQUOS sense4 basic (Android 12, Adreno 618) the lobby froze the instant the + stream started, while SteamVR still received poses. The streamer logs (resolution clamp, + "refresh rate not supported, using 60.000004Hz", a GLES `FreeAllocationOnTimestamp` message) were + harmless. The stock ALVR OpenXR client worked on the same phone. +- **Finding:** the hardware decoder started and consumed about 60 frames a second, but + `alvr_get_frame()` never returned a frame, on every render call. With ARCore turned off + (`initializeNative(..., false)`) video appeared at once. A software-decoder test was not + conclusive: on that phone the software decoder could not be created and the call to create it + hung. +- **Cause (not fully identified):** the ARCore camera session and the video decoder do not work + together when the camera starts first. The exact shared resource was not determined. +- **Fix:** `ARCORE_START_AFTER_VIDEO` (default `true`). ARCore is not resumed when the app resumes; + it is started after `ARCORE_START_AFTER_VIDEO_FRAMES` (10) decoded frames have arrived, and it is + paused again when the stream stops, so the camera is off when the next decoder is created. + Verified on the Oppo Reno 9 A and the AQUOS. Expect ARCore tracking to restart (and the position + to jump slightly) at each stream start. + +### 4.5 PowerVR GE8300 driver crash when GL debug output is on + +- **Symptom:** on a UMIDIGI A3S (Android 10, MediaTek, PowerVR Rogue GE8300, OpenGL ES 3.2 + build 1.11) the app crashed with SIGSEGV on the first lobby draw, with or without ARCore. + The old v20 client, which drew with its own C++ renderer, worked on this phone. +- **Crash location:** inside the vendor driver (`libGLESv2_mtk.so`), called from `glTexSubImage2D` + for a plain 1024x1024 RGBA8 upload from a GL buffer, in the driver's own `vsnprintf`/`strlen` + with a garbage pointer (the fault address was the texture's width and height packed together). +- **Cause:** wgpu-hal enabled GL debug output (`DEBUG_OUTPUT` plus a debug callback) and a debug EGL + context. This driver crashes while formatting one of its own debug messages. Changing the pixel + store row length / image height for the upload made no difference. +- **Fix:** in the patched `wgpu-hal` (`src/gles/egl.rs`), `false &&` guards disable both the GL + debug output and the debug EGL context. No functionality is lost; only driver diagnostics + (such as the `[Other/Performance]` messages seen on Adreno) disappear from the log. Verified on + the A3S and on the other two phones. + ## 5. Setting up the ALVR side 1. Check out ALVR at the commit above and build as usual (`prepare-alvr-deps.sh`, which runs @@ -114,17 +157,26 @@ differences. The Gradle `buildClientLib` task regenerates this file on every bui ## 6. Known issues and not done yet -- **ARCore and drift correction:** not integrated on this branch. `getPose()` in `alvr_main.cpp` - is the extension point; it currently returns Cardboard's gyro-only orientation with position - fixed at the origin. -- **Stuttering:** present in v20 as well. A fix exists on the author's separate v20 ARCore branch - and has not been merged here. Weak Wi-Fi also contributed in testing. +- **ARCore:** integrated (see `ARCORE_NOTES.md` for the design and tuning constants). Not yet done: + per-phone calibration of `ARCORE_TO_CARDBOARD` (a fixed value measured on one phone), trying a + newer ARCore than 1.46.0, pinning the ARCore C headers (currently downloaded from `main`) to the + tag matching the library, adding `` + to the manifest, and checking how the height setting feels in a headset. +- **Stuttering:** fixed. When no new decoded frame is available the client draws the previous + frame again, reprojected to the current head pose. +- **Diagnostics left in the code:** `DIAG stream` log lines in the stream branch of `alvr_main.cpp` + and the `ALVR GLES` log lines in the `wgpu-hal` patch can be removed before a release. - **Stale decoder-config event:** a `DECODER_CONFIG` event arrives after each `STREAMING_STOPPED`, and the handler creates a decoder while no stream is active. Harmless so far, but it should be tidied. +- **Software decoder:** `FORCE_SOFTWARE_DECODER` is a compile-time switch, left `false`. On at least + one Android 12 phone the software decoder ALVR asks for cannot be created and the call to create + it hangs, so do not rely on it as a fallback. - **Patch path:** the `wgpu-hal` patch is referenced by an absolute path on the development - machine, which must be replaced before others can build this. + machine in ALVR's `Cargo.toml`, which must be replaced before others can build this. - **CI:** `.github/workflows/build.yml` and `release.yml` still reference the removed `gvr` / `noGvr` Gradle tasks and will need simplifying before CI can pass. +- **Upstream:** the wgpu-hal EGL share-context bug (section 4.1) is worth reporting to the ALVR + developers. - **Likely obsolete:** `driver/PVRServer` and `code/windows` (the PC side of the legacy protocol). Left untouched. From 08e471ab445b1d7be12fd0d91e1682c4258e4941 Mon Sep 17 00:00:00 2001 From: Rob Oates Date: Sat, 10 Oct 2026 20:33:39 +0900 Subject: [PATCH 18/29] README: document ALVR v21 port, ARCore tracking, tested phones, build steps --- README.md | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+) diff --git a/README.md b/README.md index b192f8c5..55a63184 100644 --- a/README.md +++ b/README.md @@ -22,11 +22,47 @@ Use Steam VR-enabled applications with your phone as VR HMD (*Head-mounted displ | Issues | [`Github Issue Tracker`](https://github.com/PhoneVR-Developers/PhoneVR/issues) | | Discuss/Chat/Help/Real-Time Updates/Any - Discord | [`Discord Server Invite Link`](https://discord.com/invite/pNYtBNk) | +## ALVR v21 port with ARCore position tracking (this branch) + +This branch ports the Android client from ALVR v20 to ALVR v21, because newer SteamVR releases +are breaking v20 servers, and adds 6DoF position tracking with ARCore. + +* **ALVR v21:** the client talks to an ALVR **v21** streamer. v21 is not a tagged release; it is the + `master` branch of [ALVR](https://github.com/alvr-org/ALVR) (the newest tag is still v20.14.1). + This client was built against `21.0.0-dev14` (ALVR commit `9f118394`). **v20 servers are not + compatible with this branch.** +* **Video:** hardware decoding for H.264 and HEVC, and the frame stutter present in the v20 client is + fixed (when no new frame has arrived, the previous frame is shown again, re-oriented to the current + head pose). +* **Position tracking (6DoF):** on phones with ARCore, head *position* comes from ARCore and + Cardboard's rotation drift (yaw) is corrected against it with a complementary filter. Phones + without ARCore (or without Google Play Services for AR) run as 3DoF with the Cardboard tracker. + See [`ARCORE_NOTES.md`](./ARCORE_NOTES.md). +* **Status:** works, but not released yet. There are no APKs for this branch; you build it from source + (see [Building the v21 client](#building-the-v21-client)). The legacy PhoneVR server and the + `gvr` build flavour are not part of this branch. +* **Technical details and the problems found on the way:** [`MIGRATION_NOTES.md`](./MIGRATION_NOTES.md). + +Tested phones: + +| Phone | Android | GPU | ARCore | Result | +| --- | --- | --- | --- | --- | +| Oppo Reno 9 A | 14 | Adreno 619 | yes | Video, 6DoF position and drift-free yaw | +| AQUOS sense4 basic | 12 | Adreno 618 | yes | Works (ARCore is started only after video is flowing) | +| UMIDIGI A3S | 10 | PowerVR Rogue GE8300 | no | Works as 3DoF | + +Known limits: the yaw correction uses one fixed phone-to-ARCore alignment measured on a single phone, +so other phones may need a calibration that is not configurable yet; the floor height setting has not +been verified in a headset; and ARCore tracking is started again each time a stream starts, which +may cause a small position jump at that moment. + ## Contents +* [ALVR v21 port with ARCore (this branch)](#alvr-v21-port-with-arcore-position-tracking-this-branch) * [Requirements](#requirements) * [Installation](#installation) * [Advanced Configuration](#advanced-configuration) * [Development](#development) +* [Building the v21 client](#building-the-v21-client) * [Troubleshooting](#troubleshooting) * [Issue / Bug Reporting](#issue--bug-reporting) @@ -34,6 +70,8 @@ Use Steam VR-enabled applications with your phone as VR HMD (*Head-mounted displ A PC with *Windows 7 or above*, A smartphone with *Android 5.0 (Lollipop) or above* (_ALVR requires Android 8.0 or above_) with *OpenGL-ES 3.0 or above*, Steam and some SteamVR applications installed. +**For the ALVR v21 branch:** an Android 8.0+ phone with OpenGL ES 3.0+ and a hardware video decoder, and an ALVR v21 streamer on the PC. Position tracking additionally needs a phone [supported by ARCore](https://developers.google.com/ar/devices) with *Google Play Services for AR* installed; without it the app still runs, as 3DoF. + ## Installation PhoneVR(PVR) can be used with two servers ALVR and PVRServer. ALVR Server is recommended over PVRServer, both use different APIs. @@ -56,6 +94,8 @@ PhoneVR(PVR) can be used with two servers ALVR and PVRServer. ALVR Server is rec | Code backend | Rust + C++ | C++ | ### A. Installation using ALVR Server +> **This branch (v21):** use an ALVR **v21** streamer, i.e. a build from ALVR's `master` branch (tested with `21.0.0-dev14`); the v20.8.0 mentioned below does not work with it. The older-version list below applies to the v20 releases of PhoneVR. + * Currently only **[ALVR v20.8.0](https://github.com/alvr-org/ALVR/releases/tag/v20.8.0)** (latest stable) is supported by PhoneVR. * Older version - PhoneVR v1.4.0-beta -> ALVR v20.7.1 @@ -66,6 +106,8 @@ PhoneVR(PVR) can be used with two servers ALVR and PVRServer. ALVR Server is rec * Use the install guide of ALVR Server from [here](https://github.com/alvr-org/ALVR#install). ### B. Installation using Legacy PhoneVR Server +> Not available on the ALVR v21 branch. + * Make sure you have Steam and SteamVR installed (To find SteamVR on steam, `Library -> Tools -> SteamVR`). * Download latest release [`PhoneVR-Server-vX.X.X-xxxxx.zip`](https://github.com/PhoneVR-Developers/PhoneVR/releases/latest) of this repository. * Copy the whole folder `PVRServer` in `driver` folder of zip file into your `SteamVR/drivers` folder. (Default Path: `C:\Program Files (x86)\Steam\steamapps\common\SteamVR\drivers`. @@ -74,6 +116,7 @@ PhoneVR(PVR) can be used with two servers ALVR and PVRServer. ALVR Server is rec ### Common Instructions for both Servers * Install the Android Apk on your mobile from [`Releases`](https://github.com/PhoneVR-Developers/PhoneVR/releases/latest) page. * There are two Apks on the release page, one with GoogleVR(GVR) SDK and other without GVR SDK. If you plan to use ALVR-Server only you can install `PhoneVR-vX.X.X.noGvr.apk`. + * The ALVR v21 branch has no release APKs yet; build it as described in [Building the v21 client](#building-the-v21-client). * Make sure that "Run in Background", "Auto Start"(Restart on Crash) permissions, if exists on your device(especially Xiaomi users), are given. Also make sure that any kind of 3rd party battery saver app dosen't kill PhoneVR when in background. * After opening the installed app, Choose ALVR(recommended) or PhoneVR Server, according to the server you installed. @@ -142,6 +185,7 @@ This Project is presently under testing. But, pull requests are welcome. * Android App: App folder: `/code/mobile/android/PhoneVR` * Compiled/Tested on `Android Studio 4.0.1` * For testing, this project has 2 Build Configs, Debug and Release. Debug has lots of debugging callouts to logcat from both JAVA and JNI. + * (The ALVR v21 branch builds a single ALVR-based variant; the gvr/noGvr split described next applies to the v20 line.) * There are also two build flavours: gvr and noGvr. The gvr flavour is a superset of the noGvr flavour that adds support for legacy PhoneVR server with Google VR SDK (deprecated). * The source code of the gvr flavour is in app/src/gvr/ and app/src/main/. The code of the noGvr flavour is in app/src/noGvr/ and app/src/main/. Respectively, any shared code in app/src/main/ has to be tested with both flavours. * The prepare-alvr-deps.sh script, which needs to be run before building supports the optional "nogvr" argument. If you run the script without arguments, you can build both flavours. If you run the script with the "nogvr" argument, you can only build the noGvr flavour. @@ -163,13 +207,44 @@ This Project is presently under testing. But, pull requests are welcome. * GoogleVR 1.200 (https://github.com/googlevr/gvr-android-sdk) (code\mobile\android\libraries\jni) * ACRA v5.7 (https://github.com/ACRA/acra) * ALVR (https://github.com/alvr-org/ALVR) (code\mobile\android\PhoneVR\ALVR) + * ARCore SDK 1.46.0 (https://github.com/google-ar/arcore-android-sdk) (ALVR v21 branch; C headers downloaded by `prepare-alvr-deps.sh`) + * wgpu-hal 25.0.2, patched (https://github.com/gfx-rs/wgpu) (ALVR v21 branch; patch in `code/mobile/android/PhoneVR/patches/`) * Code styling * Repo enforces clang-format v16.0.0 (.clang-format) for C/C++ and spotless (.editorconfig, google-java-format, ktfmt) for Java/Kotlin files * To autofix clang-format'ing issues devs can use `make format` * To autofix spotless format issues devs can goto PhoneVR android project folder and use `./gradlew spotlessApply` or use [spotless-gradle](https://plugins.jetbrains.com/plugin/18321-spotless-gradle) plugin for AndroidStudio/IDEs +## Building the v21 client + +Tested on Ubuntu (a VirtualBox VM) with JDK 17, Rust 1.97.1 and Android NDK 25.2.9519653. These steps +are condensed from [`MIGRATION_NOTES.md`](./MIGRATION_NOTES.md), which has the details. + +1. Install the Android SDK (a platform and build-tools must be installed) and NDK, and set + `JAVA_HOME`, `ANDROID_HOME` and `ANDROID_NDK_HOME`. +2. In `code/mobile/android/PhoneVR`, run `prepare-alvr-deps.sh`. It prepares the ALVR dependencies + (including `cargo xtask build-client-lib`) and downloads the ARCore C headers. +3. Patch `wgpu-hal` 25.0.2. Without this patch the lobby and video do not render, and some phones + (for example one with a PowerVR GE8300 GPU) crash. Download the crate from + `https://static.crates.io/crates/wgpu-hal/wgpu-hal-25.0.2.crate`, unpack it, and from inside the + unpacked folder run + `patch -p1 < /code/mobile/android/PhoneVR/patches/wgpu-hal-25.0.2-share-egl-context.patch`. + It makes the Rust GL context share the app's EGL context and turns off GL debug output. +4. In ALVR's top-level `Cargo.toml`, point `[patch.crates-io]` at the patched folder: + `wgpu-hal = { path = "" }`. + (The path used during development was absolute and machine-specific; use your own.) +5. Build the app with Gradle (Android Studio or `./gradlew assembleDebug`). + +To test without ARCore, pass `false` as the last argument of `initializeNative` in `ALVRActivity.java`. + ## Troubleshooting +* **(ALVR v21 branch)** The lobby freezes as soon as the stream starts, while SteamVR still shows head movement + - This happened on phones where the ARCore camera and the video decoder conflict. This branch starts ARCore only after video is flowing (`ARCORE_START_AFTER_VIDEO` in `alvr_main.cpp`). If you still see it, try turning ARCore off (last argument of `initializeNative` in `ALVRActivity.java`) and send the log. +* **(ALVR v21 branch)** The app crashes right after pressing the start button + - Capture `adb logcat -d | grep -a -E "FATAL|UnsatisfiedLinkError|Fatal signal|panicked|dlopen"` and check that the `wgpu-hal` patch (see [Building the v21 client](#building-the-v21-client)) was applied. +* **(ALVR v21 branch)** No position tracking + - Check the log for `ARCore: tracking`. If you see `could not create session`, the phone does not support ARCore or Google Play Services for AR is missing; the app then continues as 3DoF. + * Android App doesn’t connect to Windows Steam VR even after opening Phone App first and then SteamVR on windows 1. Make sure that both the Windows and Android devices are in the same Local Network. (connected to the same router/wifi device) 2. Sometimes, the port which PhoneVR use to connect win/mobile devices, also known as `Pairing Port (default :33333)`, might be used by other services on your devices(Windows/Android). Try changing the "Pairing Port" on `Android PhoneVR App` settings and `pairing_port` in `C:\Program Files\PhoneVR\pvrsettings.json` for `Windows PVR driver` and restart SteamVR. **Both Windows Pairing port and Android Pairing port should be the same**. Safe recommended port range : `30000 - 65535` From 075cadc974f1391fb9d57b9bb7e2080281bdfa23 Mon Sep 17 00:00:00 2001 From: Rob Oates Date: Tue, 13 Oct 2026 20:57:05 +0900 Subject: [PATCH 19/29] Cleanup: remove DIAG logging and old decoder block; trim wgpu-hal patch to the real changes --- .../PhoneVR/app/src/main/cpp/alvr_main.cpp | 51 +- .../wgpu-hal-25.0.2-share-egl-context.patch | 1644 +---------------- 2 files changed, 11 insertions(+), 1684 deletions(-) diff --git a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp index 0262bf5a..f88cff06 100644 --- a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp +++ b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp @@ -1112,36 +1112,13 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render * separate event, ALVR_EVENT_DECODER_CONFIG, which is guaranteed to arrive before * the first frame. */ - /* - if (!g_decoderCreated) { - auto configSize = alvr_get_decoder_config(nullptr); - auto configBuffer = std::vector(configSize); - alvr_get_decoder_config(configBuffer.data()); - - AlvrDecoderConfig decoderConfig = {}; - decoderConfig.codec = event.DECODER_CONFIG.codec; - decoderConfig.force_software_decoder = true; - decoderConfig.max_buffering_frames = 1.5f; - decoderConfig.buffering_history_weight = 0.9f; - decoderConfig.options = nullptr; - decoderConfig.options_count = 0; - decoderConfig.config_buffer = - reinterpret_cast(configBuffer.data()); - decoderConfig.config_buffer_size = configSize; - - info("ALVR Poll Event: ALVR_EVENT_DECODER_CONFIG, codec=%d, config bytes=%llu", - (int) decoderConfig.codec, (unsigned long long) configSize); - alvr_create_decoder(decoderConfig); - g_decoderCreated = true; - } - */ - if (!g_decoderCreated) { auto configSize = alvr_get_decoder_config(nullptr); auto configBuffer = std::vector(configSize); alvr_get_decoder_config(configBuffer.data()); - // false = try the phone's hardware decoder; flip to true if it fails again. + // false = use the phone's hardware decoder. true forces ALVR's software decoder, + // which cannot be created on some phones (creating it then hangs), so leave it false. constexpr bool FORCE_SOFTWARE_DECODER = false; // Same options the stock ALVR client gets from the server's defaults. @@ -1226,11 +1203,6 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render // below. bool isNewFrame = alvr_get_frame(×tampNs, &streamHardwareBuffer); - // DIAGNOSTIC (phone-1 freeze): count what the stream branch does, log once a second. - static uint64_t diagLastNs = 0, diagCalls = 0, diagNew = 0, diagNoMeta = 0, diagRendered = 0; - diagCalls++; - if (isNewFrame) diagNew++; - AlvrVideoFrameMetadata metadata; if (isNewFrame) { if (ARCORE_START_AFTER_VIDEO && CTX.videoFramesSeen < ARCORE_START_AFTER_VIDEO_FRAMES && @@ -1251,7 +1223,6 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render metadata = CTX.lastFrameMetadata; } else { // No metadata at all yet (first frame) - nothing sensible to reproject against. - diagNoMeta++; return; } } else if (CTX.lastStreamBuffer != nullptr && CTX.hasFrameMetadata) { @@ -1261,12 +1232,6 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render metadata = CTX.lastFrameMetadata; } else { // Nothing has been decoded yet. - { - static uint64_t lastNs = 0; uint64_t n = GetBootTimeNano(); - if (n - lastNs > 1000000000ULL) { lastNs = n; - __android_log_print(ANDROID_LOG_INFO, "ALVR_PVR_NATIVE", - "DIAG stream: no decoded frame yet (alvr_get_frame returned false, calls=%llu)", (unsigned long long)diagCalls); } - } return; } @@ -1292,18 +1257,6 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render ALVR_GL_CALL( alvr_render_stream_opengl(streamHardwareBuffer, streamViewParams, foveationCenters)); EGLCHK("after alvr_render_stream_opengl()"); - diagRendered++; - { - uint64_t nowNs = GetBootTimeNano(); - if (nowNs - diagLastNs > 1000000000ULL) { - __android_log_print(ANDROID_LOG_INFO, "ALVR_PVR_NATIVE", - "DIAG stream: calls=%llu newFrames=%llu noMetadataReturns=%llu rendered=%llu lastTs=%llu glError=0x%x", - (unsigned long long)diagCalls, (unsigned long long)diagNew, - (unsigned long long)diagNoMeta, (unsigned long long)diagRendered, - (unsigned long long)timestampNs, glGetError()); - diagLastNs = nowNs; diagCalls = diagNew = diagNoMeta = diagRendered = 0; - } - } // The statistics are per decoded frame, so only report frames that are new. if (isNewFrame) { diff --git a/code/mobile/android/PhoneVR/patches/wgpu-hal-25.0.2-share-egl-context.patch b/code/mobile/android/PhoneVR/patches/wgpu-hal-25.0.2-share-egl-context.patch index 10505be4..25f43fe0 100644 --- a/code/mobile/android/PhoneVR/patches/wgpu-hal-25.0.2-share-egl-context.patch +++ b/code/mobile/android/PhoneVR/patches/wgpu-hal-25.0.2-share-egl-context.patch @@ -1,31 +1,7 @@ -diff -ruN a/src/gles/egl.rs b/src/gles/egl.rs ---- a/src/gles/egl.rs 2026-10-10 19:59:35.659395136 +0900 -+++ b/src/gles/egl.rs 2026-10-10 19:59:35.673210214 +0900 -@@ -410,9 +410,13 @@ - impl<'a> Drop for AdapterContextLock<'a> { - fn drop(&mut self) { - if let Some(egl) = self.egl.take() { -+ log::info!("ALVR GLES UNLOCK: releasing wgpu EGL context"); -+ - egl.instance - .make_current(egl.display, None, None, None) - .unwrap(); -+ -+ log::info!("ALVR GLES UNLOCK: wgpu EGL context released"); - } - } - } -@@ -449,7 +453,9 @@ - .expect("Could not lock adapter context. This is most-likely a deadlock."); - - let egl = self.egl.as_ref().map(|egl| { -+ log::info!("ALVR GLES LOCK: making wgpu EGL context current"); - egl.make_current(); -+ log::info!("ALVR GLES LOCK: wgpu EGL context is current"); - EglContextLock { - instance: &egl.instance, - display: egl.display, -@@ -615,7 +621,8 @@ +diff -ruN -x '*.bak' -x '*.orig' -x '*.rej' a/src/gles/egl.rs b/src/gles/egl.rs +--- a/src/gles/egl.rs ++++ b/src/gles/egl.rs +@@ -615,7 +615,8 @@ wgt::Gles3MinorVersion::Version2 => 2, }); } @@ -35,18 +11,16 @@ diff -ruN a/src/gles/egl.rs b/src/gles/egl.rs if version >= (1, 5) { log::debug!("\tEGL context: +debug"); context_attributes.push(khronos_egl::CONTEXT_OPENGL_DEBUG); -@@ -654,11 +661,16 @@ +@@ -654,11 +655,15 @@ gl_context_attributes.extend(&context_attributes); gles_context_attributes.extend(&context_attributes); -- let context = if supports_opengl { -- egl.create_context(display, config, None, &gl_context_attributes) + // ALVR: share GL objects with the app's current EGL context, so textures created by the + // app (glGenTextures) are visible inside this context. + let share_context = egl.get_current_context(); -+ log::info!("ALVR GLES: share context = {:?}", share_context.map(|c| c.as_ptr())); + -+ let context = if supports_opengl { + let context = if supports_opengl { +- egl.create_context(display, config, None, &gl_context_attributes) + egl.create_context(display, config, share_context, &gl_context_attributes) .or_else(|_| { egl.bind_api(khronos_egl::OPENGL_ES_API).unwrap(); @@ -55,7 +29,7 @@ diff -ruN a/src/gles/egl.rs b/src/gles/egl.rs }) .map_err(|e| { crate::InstanceError::with_source( -@@ -667,7 +679,7 @@ +@@ -667,7 +672,7 @@ ) }) } else { @@ -64,7 +38,7 @@ diff -ruN a/src/gles/egl.rs b/src/gles/egl.rs .map_err(|e| { crate::InstanceError::with_source( String::from("unable to create GLES 3.x context"), -@@ -1099,7 +1111,9 @@ +@@ -1099,7 +1104,9 @@ }); } @@ -75,1603 +49,3 @@ diff -ruN a/src/gles/egl.rs b/src/gles/egl.rs log::debug!("Enabling GLES debug output"); unsafe { gl.enable(glow::DEBUG_OUTPUT) }; unsafe { gl.debug_message_callback(super::gl_debug_message_callback) }; -diff -ruN a/src/gles/egl.rs.bak b/src/gles/egl.rs.bak ---- a/src/gles/egl.rs.bak 1970-01-01 09:00:00.000000000 +0900 -+++ b/src/gles/egl.rs.bak 2026-10-10 19:59:35.674178122 +0900 -@@ -0,0 +1,1559 @@ -+#![allow(clippy::std_instead_of_alloc, clippy::std_instead_of_core)] -+ -+use std::{ -+ ffi, -+ mem::ManuallyDrop, -+ os::raw, -+ ptr, -+ rc::Rc, -+ string::String, -+ sync::{Arc, LazyLock}, -+ time::Duration, -+ vec::Vec, -+}; -+ -+use glow::HasContext; -+use hashbrown::HashMap; -+use parking_lot::{MappedMutexGuard, Mutex, MutexGuard, RwLock}; -+ -+/// The amount of time to wait while trying to obtain a lock to the adapter context -+const CONTEXT_LOCK_TIMEOUT_SECS: u64 = 1; -+ -+const EGL_CONTEXT_FLAGS_KHR: i32 = 0x30FC; -+const EGL_CONTEXT_OPENGL_DEBUG_BIT_KHR: i32 = 0x0001; -+const EGL_CONTEXT_OPENGL_ROBUST_ACCESS_EXT: i32 = 0x30BF; -+const EGL_PLATFORM_WAYLAND_KHR: u32 = 0x31D8; -+const EGL_PLATFORM_X11_KHR: u32 = 0x31D5; -+const EGL_PLATFORM_ANGLE_ANGLE: u32 = 0x3202; -+const EGL_PLATFORM_ANGLE_NATIVE_PLATFORM_TYPE_ANGLE: u32 = 0x348F; -+const EGL_PLATFORM_ANGLE_DEBUG_LAYERS_ENABLED: u32 = 0x3451; -+const EGL_PLATFORM_SURFACELESS_MESA: u32 = 0x31DD; -+const EGL_GL_COLORSPACE_KHR: u32 = 0x309D; -+const EGL_GL_COLORSPACE_SRGB_KHR: u32 = 0x3089; -+ -+type XOpenDisplayFun = -+ unsafe extern "system" fn(display_name: *const raw::c_char) -> *mut raw::c_void; -+ -+type XCloseDisplayFun = unsafe extern "system" fn(display: *mut raw::c_void) -> raw::c_int; -+ -+type WlDisplayConnectFun = -+ unsafe extern "system" fn(display_name: *const raw::c_char) -> *mut raw::c_void; -+ -+type WlDisplayDisconnectFun = unsafe extern "system" fn(display: *const raw::c_void); -+ -+#[cfg(not(Emscripten))] -+type EglInstance = khronos_egl::DynamicInstance; -+ -+#[cfg(Emscripten)] -+type EglInstance = khronos_egl::Instance; -+ -+type WlEglWindowCreateFun = unsafe extern "system" fn( -+ surface: *const raw::c_void, -+ width: raw::c_int, -+ height: raw::c_int, -+) -> *mut raw::c_void; -+ -+type WlEglWindowResizeFun = unsafe extern "system" fn( -+ window: *const raw::c_void, -+ width: raw::c_int, -+ height: raw::c_int, -+ dx: raw::c_int, -+ dy: raw::c_int, -+); -+ -+type WlEglWindowDestroyFun = unsafe extern "system" fn(window: *const raw::c_void); -+ -+type EglLabel = *const raw::c_void; -+ -+#[allow(clippy::upper_case_acronyms)] -+type EGLDEBUGPROCKHR = Option< -+ unsafe extern "system" fn( -+ error: khronos_egl::Enum, -+ command: *const raw::c_char, -+ message_type: u32, -+ thread_label: EglLabel, -+ object_label: EglLabel, -+ message: *const raw::c_char, -+ ), -+>; -+ -+const EGL_DEBUG_MSG_CRITICAL_KHR: u32 = 0x33B9; -+const EGL_DEBUG_MSG_ERROR_KHR: u32 = 0x33BA; -+const EGL_DEBUG_MSG_WARN_KHR: u32 = 0x33BB; -+const EGL_DEBUG_MSG_INFO_KHR: u32 = 0x33BC; -+ -+type EglDebugMessageControlFun = unsafe extern "system" fn( -+ proc: EGLDEBUGPROCKHR, -+ attrib_list: *const khronos_egl::Attrib, -+) -> raw::c_int; -+ -+unsafe extern "system" fn egl_debug_proc( -+ error: khronos_egl::Enum, -+ command_raw: *const raw::c_char, -+ message_type: u32, -+ _thread_label: EglLabel, -+ _object_label: EglLabel, -+ message_raw: *const raw::c_char, -+) { -+ let log_severity = match message_type { -+ EGL_DEBUG_MSG_CRITICAL_KHR | EGL_DEBUG_MSG_ERROR_KHR => log::Level::Error, -+ EGL_DEBUG_MSG_WARN_KHR => log::Level::Warn, -+ EGL_DEBUG_MSG_INFO_KHR => log::Level::Info, -+ _ => log::Level::Debug, -+ }; -+ let command = unsafe { ffi::CStr::from_ptr(command_raw) }.to_string_lossy(); -+ let message = if message_raw.is_null() { -+ "".into() -+ } else { -+ unsafe { ffi::CStr::from_ptr(message_raw) }.to_string_lossy() -+ }; -+ -+ log::log!( -+ log_severity, -+ "EGL '{}' code 0x{:x}: {}", -+ command, -+ error, -+ message, -+ ); -+} -+ -+/// A simple wrapper around an X11 or Wayland display handle. -+/// Since the logic in this file doesn't actually need to directly -+/// persist a wayland connection handle, the only load-bearing -+/// enum variant is the X11 variant -+#[derive(Debug)] -+enum DisplayRef { -+ X11(ptr::NonNull), -+ Wayland, -+} -+ -+impl DisplayRef { -+ /// Convenience for getting the underlying pointer -+ fn as_ptr(&self) -> *mut raw::c_void { -+ match *self { -+ Self::X11(ptr) => ptr.as_ptr(), -+ Self::Wayland => unreachable!(), -+ } -+ } -+} -+ -+/// DisplayOwner ties the lifetime of the system display handle -+/// to that of the loaded library. -+/// It implements Drop to ensure that the display handle is closed -+/// prior to unloading the library so that we don't leak the -+/// associated file descriptors -+#[derive(Debug)] -+struct DisplayOwner { -+ library: libloading::Library, -+ display: DisplayRef, -+} -+ -+impl Drop for DisplayOwner { -+ fn drop(&mut self) { -+ match self.display { -+ DisplayRef::X11(ptr) => unsafe { -+ let func: libloading::Symbol = -+ self.library.get(c"XCloseDisplay".to_bytes()).unwrap(); -+ func(ptr.as_ptr()); -+ }, -+ DisplayRef::Wayland => {} -+ } -+ } -+} -+ -+fn open_x_display() -> Option { -+ log::debug!("Loading X11 library to get the current display"); -+ unsafe { -+ let library = find_library(&["libX11.so.6", "libX11.so"])?; -+ let func: libloading::Symbol = -+ library.get(c"XOpenDisplay".to_bytes()).unwrap(); -+ let result = func(ptr::null()); -+ ptr::NonNull::new(result).map(|ptr| DisplayOwner { -+ display: DisplayRef::X11(ptr), -+ library, -+ }) -+ } -+} -+ -+unsafe fn find_library(paths: &[&str]) -> Option { -+ for path in paths { -+ match unsafe { libloading::Library::new(path) } { -+ Ok(lib) => return Some(lib), -+ _ => continue, -+ }; -+ } -+ None -+} -+ -+fn test_wayland_display() -> Option { -+ /* We try to connect and disconnect here to simply ensure there -+ * is an active wayland display available. -+ */ -+ log::debug!("Loading Wayland library to get the current display"); -+ let library = unsafe { -+ let client_library = find_library(&["libwayland-client.so.0", "libwayland-client.so"])?; -+ let wl_display_connect: libloading::Symbol = client_library -+ .get(c"wl_display_connect".to_bytes()) -+ .unwrap(); -+ let wl_display_disconnect: libloading::Symbol = client_library -+ .get(c"wl_display_disconnect".to_bytes()) -+ .unwrap(); -+ let display = ptr::NonNull::new(wl_display_connect(ptr::null()))?; -+ wl_display_disconnect(display.as_ptr()); -+ find_library(&["libwayland-egl.so.1", "libwayland-egl.so"])? -+ }; -+ Some(DisplayOwner { -+ library, -+ display: DisplayRef::Wayland, -+ }) -+} -+ -+#[derive(Clone, Copy, Debug)] -+enum SrgbFrameBufferKind { -+ /// No support for SRGB surface -+ None, -+ /// Using EGL 1.5's support for colorspaces -+ Core, -+ /// Using EGL_KHR_gl_colorspace -+ Khr, -+} -+ -+/// Choose GLES framebuffer configuration. -+fn choose_config( -+ egl: &EglInstance, -+ display: khronos_egl::Display, -+ srgb_kind: SrgbFrameBufferKind, -+) -> Result<(khronos_egl::Config, bool), crate::InstanceError> { -+ //TODO: EGL_SLOW_CONFIG -+ let tiers = [ -+ ( -+ "off-screen", -+ &[ -+ khronos_egl::SURFACE_TYPE, -+ khronos_egl::PBUFFER_BIT, -+ khronos_egl::RENDERABLE_TYPE, -+ khronos_egl::OPENGL_ES2_BIT, -+ ][..], -+ ), -+ ( -+ "presentation", -+ &[khronos_egl::SURFACE_TYPE, khronos_egl::WINDOW_BIT][..], -+ ), -+ #[cfg(not(target_os = "android"))] -+ ( -+ "native-render", -+ &[khronos_egl::NATIVE_RENDERABLE, khronos_egl::TRUE as _][..], -+ ), -+ ]; -+ -+ let mut attributes = Vec::with_capacity(9); -+ for tier_max in (0..tiers.len()).rev() { -+ let name = tiers[tier_max].0; -+ log::debug!("\tTrying {}", name); -+ -+ attributes.clear(); -+ for &(_, tier_attr) in tiers[..=tier_max].iter() { -+ attributes.extend_from_slice(tier_attr); -+ } -+ // make sure the Alpha is enough to support sRGB -+ match srgb_kind { -+ SrgbFrameBufferKind::None => {} -+ _ => { -+ attributes.push(khronos_egl::ALPHA_SIZE); -+ attributes.push(8); -+ } -+ } -+ attributes.push(khronos_egl::NONE); -+ -+ match egl.choose_first_config(display, &attributes) { -+ Ok(Some(config)) => { -+ if tier_max == 1 { -+ //Note: this has been confirmed to malfunction on Intel+NV laptops, -+ // but also on Angle. -+ log::warn!("EGL says it can present to the window but not natively",); -+ } -+ // Android emulator can't natively present either. -+ let tier_threshold = -+ if cfg!(target_os = "android") || cfg!(windows) || cfg!(target_env = "ohos") { -+ 1 -+ } else { -+ 2 -+ }; -+ return Ok((config, tier_max >= tier_threshold)); -+ } -+ Ok(None) => { -+ log::warn!("No config found!"); -+ } -+ Err(e) => { -+ log::error!("error in choose_first_config: {:?}", e); -+ } -+ } -+ } -+ -+ // TODO: include diagnostic details that are currently logged -+ Err(crate::InstanceError::new(String::from( -+ "unable to find an acceptable EGL framebuffer configuration", -+ ))) -+} -+ -+#[derive(Clone, Debug)] -+struct EglContext { -+ instance: Arc, -+ version: (i32, i32), -+ display: khronos_egl::Display, -+ raw: khronos_egl::Context, -+ pbuffer: Option, -+} -+ -+impl EglContext { -+ fn make_current(&self) { -+ self.instance -+ .make_current(self.display, self.pbuffer, self.pbuffer, Some(self.raw)) -+ .unwrap(); -+ } -+ -+ fn unmake_current(&self) { -+ self.instance -+ .make_current(self.display, None, None, None) -+ .unwrap(); -+ } -+} -+ -+/// A wrapper around a [`glow::Context`] and the required EGL context that uses locking to guarantee -+/// exclusive access when shared with multiple threads. -+pub struct AdapterContext { -+ glow: Mutex>, -+ egl: Option, -+} -+ -+unsafe impl Sync for AdapterContext {} -+unsafe impl Send for AdapterContext {} -+ -+impl AdapterContext { -+ pub fn is_owned(&self) -> bool { -+ self.egl.is_some() -+ } -+ -+ /// Returns the EGL instance. -+ /// -+ /// This provides access to EGL functions and the ability to load GL and EGL extension functions. -+ pub fn egl_instance(&self) -> Option<&EglInstance> { -+ self.egl.as_ref().map(|egl| &*egl.instance) -+ } -+ -+ /// Returns the EGLDisplay corresponding to the adapter context. -+ /// -+ /// Returns [`None`] if the adapter was externally created. -+ pub fn raw_display(&self) -> Option<&khronos_egl::Display> { -+ self.egl.as_ref().map(|egl| &egl.display) -+ } -+ -+ /// Returns the EGL version the adapter context was created with. -+ /// -+ /// Returns [`None`] if the adapter was externally created. -+ pub fn egl_version(&self) -> Option<(i32, i32)> { -+ self.egl.as_ref().map(|egl| egl.version) -+ } -+ -+ pub fn raw_context(&self) -> *mut raw::c_void { -+ match self.egl { -+ Some(ref egl) => egl.raw.as_ptr(), -+ None => ptr::null_mut(), -+ } -+ } -+} -+ -+impl Drop for AdapterContext { -+ fn drop(&mut self) { -+ struct CurrentGuard<'a>(&'a EglContext); -+ impl Drop for CurrentGuard<'_> { -+ fn drop(&mut self) { -+ self.0.unmake_current(); -+ } -+ } -+ -+ // Context must be current when dropped. See safety docs on -+ // `glow::HasContext`. -+ // -+ // NOTE: This is only set to `None` by `Adapter::new_external` which -+ // requires the context to be current when anything that may be holding -+ // the `Arc` is dropped. -+ let _guard = self.egl.as_ref().map(|egl| { -+ egl.make_current(); -+ CurrentGuard(egl) -+ }); -+ let glow = self.glow.get_mut(); -+ // SAFETY: Field not used after this. -+ unsafe { ManuallyDrop::drop(glow) }; -+ } -+} -+ -+struct EglContextLock<'a> { -+ instance: &'a Arc, -+ display: khronos_egl::Display, -+} -+ -+/// A guard containing a lock to an [`AdapterContext`], while the GL context is kept current. -+pub struct AdapterContextLock<'a> { -+ glow: MutexGuard<'a, ManuallyDrop>, -+ egl: Option>, -+} -+ -+impl<'a> std::ops::Deref for AdapterContextLock<'a> { -+ type Target = glow::Context; -+ -+ fn deref(&self) -> &Self::Target { -+ &self.glow -+ } -+} -+ -+impl<'a> Drop for AdapterContextLock<'a> { -+ fn drop(&mut self) { -+ if let Some(egl) = self.egl.take() { -+ log::info!("ALVR GLES UNLOCK: releasing wgpu EGL context"); -+ -+ egl.instance -+ .make_current(egl.display, None, None, None) -+ .unwrap(); -+ -+ log::info!("ALVR GLES UNLOCK: wgpu EGL context released"); -+ } -+ } -+} -+ -+impl AdapterContext { -+ /// Get's the [`glow::Context`] without waiting for a lock -+ /// -+ /// # Safety -+ /// -+ /// This should only be called when you have manually made sure that the current thread has made -+ /// the EGL context current and that no other thread also has the EGL context current. -+ /// Additionally, you must manually make the EGL context **not** current after you are done with -+ /// it, so that future calls to `lock()` will not fail. -+ /// -+ /// > **Note:** Calling this function **will** still lock the [`glow::Context`] which adds an -+ /// > extra safe-guard against accidental concurrent access to the context. -+ pub unsafe fn get_without_egl_lock(&self) -> MappedMutexGuard { -+ let guard = self -+ .glow -+ .try_lock_for(Duration::from_secs(CONTEXT_LOCK_TIMEOUT_SECS)) -+ .expect("Could not lock adapter context. This is most-likely a deadlock."); -+ MutexGuard::map(guard, |glow| &mut **glow) -+ } -+ -+ /// Obtain a lock to the EGL context and get handle to the [`glow::Context`] that can be used to -+ /// do rendering. -+ #[track_caller] -+ pub fn lock<'a>(&'a self) -> AdapterContextLock<'a> { -+ let glow = self -+ .glow -+ // Don't lock forever. If it takes longer than 1 second to get the lock we've got a -+ // deadlock and should panic to show where we got stuck -+ .try_lock_for(Duration::from_secs(CONTEXT_LOCK_TIMEOUT_SECS)) -+ .expect("Could not lock adapter context. This is most-likely a deadlock."); -+ -+ let egl = self.egl.as_ref().map(|egl| { -+ log::info!("ALVR GLES LOCK: making wgpu EGL context current"); -+ egl.make_current(); -+ log::info!("ALVR GLES LOCK: wgpu EGL context is current"); -+ EglContextLock { -+ instance: &egl.instance, -+ display: egl.display, -+ } -+ }); -+ -+ AdapterContextLock { glow, egl } -+ } -+} -+ -+#[derive(Debug)] -+struct Inner { -+ /// Note: the context contains a dummy pbuffer (1x1). -+ /// Required for `eglMakeCurrent` on platforms that doesn't supports `EGL_KHR_surfaceless_context`. -+ egl: EglContext, -+ #[allow(unused)] -+ version: (i32, i32), -+ supports_native_window: bool, -+ config: khronos_egl::Config, -+ #[cfg_attr(Emscripten, allow(dead_code))] -+ wl_display: Option<*mut raw::c_void>, -+ #[cfg_attr(Emscripten, allow(dead_code))] -+ force_gles_minor_version: wgt::Gles3MinorVersion, -+ /// Method by which the framebuffer should support srgb -+ srgb_kind: SrgbFrameBufferKind, -+} -+ -+// Different calls to `eglGetPlatformDisplay` may return the same `Display`, making it a global -+// state of all our `EglContext`s. This forces us to track the number of such context to prevent -+// terminating the display if it's currently used by another `EglContext`. -+static DISPLAYS_REFERENCE_COUNT: LazyLock>> = -+ LazyLock::new(Default::default); -+ -+fn initialize_display( -+ egl: &EglInstance, -+ display: khronos_egl::Display, -+) -> Result<(i32, i32), khronos_egl::Error> { -+ let mut guard = DISPLAYS_REFERENCE_COUNT.lock(); -+ *guard.entry(display.as_ptr() as usize).or_default() += 1; -+ -+ // We don't need to check the reference count here since according to the `eglInitialize` -+ // documentation, initializing an already initialized EGL display connection has no effect -+ // besides returning the version numbers. -+ egl.initialize(display) -+} -+ -+fn terminate_display( -+ egl: &EglInstance, -+ display: khronos_egl::Display, -+) -> Result<(), khronos_egl::Error> { -+ let key = &(display.as_ptr() as usize); -+ let mut guard = DISPLAYS_REFERENCE_COUNT.lock(); -+ let count_ref = guard -+ .get_mut(key) -+ .expect("Attempted to decref a display before incref was called"); -+ -+ if *count_ref > 1 { -+ *count_ref -= 1; -+ -+ Ok(()) -+ } else { -+ guard.remove(key); -+ -+ egl.terminate(display) -+ } -+} -+ -+impl Inner { -+ fn create( -+ flags: wgt::InstanceFlags, -+ egl: Arc, -+ display: khronos_egl::Display, -+ force_gles_minor_version: wgt::Gles3MinorVersion, -+ ) -> Result { -+ let version = initialize_display(&egl, display).map_err(|e| { -+ crate::InstanceError::with_source( -+ String::from("failed to initialize EGL display connection"), -+ e, -+ ) -+ })?; -+ let vendor = egl -+ .query_string(Some(display), khronos_egl::VENDOR) -+ .unwrap(); -+ let display_extensions = egl -+ .query_string(Some(display), khronos_egl::EXTENSIONS) -+ .unwrap() -+ .to_string_lossy(); -+ log::debug!("Display vendor {:?}, version {:?}", vendor, version,); -+ log::debug!( -+ "Display extensions: {:#?}", -+ display_extensions.split_whitespace().collect::>() -+ ); -+ -+ let srgb_kind = if version >= (1, 5) { -+ log::debug!("\tEGL surface: +srgb"); -+ SrgbFrameBufferKind::Core -+ } else if display_extensions.contains("EGL_KHR_gl_colorspace") { -+ log::debug!("\tEGL surface: +srgb khr"); -+ SrgbFrameBufferKind::Khr -+ } else { -+ log::warn!("\tEGL surface: -srgb"); -+ SrgbFrameBufferKind::None -+ }; -+ -+ if log::max_level() >= log::LevelFilter::Trace { -+ log::trace!("Configurations:"); -+ let config_count = egl.get_config_count(display).unwrap(); -+ let mut configurations = Vec::with_capacity(config_count); -+ egl.get_configs(display, &mut configurations).unwrap(); -+ for &config in configurations.iter() { -+ log::trace!("\tCONFORMANT=0x{:X}, RENDERABLE=0x{:X}, NATIVE_RENDERABLE=0x{:X}, SURFACE_TYPE=0x{:X}, ALPHA_SIZE={}", -+ egl.get_config_attrib(display, config, khronos_egl::CONFORMANT).unwrap(), -+ egl.get_config_attrib(display, config, khronos_egl::RENDERABLE_TYPE).unwrap(), -+ egl.get_config_attrib(display, config, khronos_egl::NATIVE_RENDERABLE).unwrap(), -+ egl.get_config_attrib(display, config, khronos_egl::SURFACE_TYPE).unwrap(), -+ egl.get_config_attrib(display, config, khronos_egl::ALPHA_SIZE).unwrap(), -+ ); -+ } -+ } -+ -+ let (config, supports_native_window) = choose_config(&egl, display, srgb_kind)?; -+ -+ let supports_opengl = if version >= (1, 4) { -+ let client_apis = egl -+ .query_string(Some(display), khronos_egl::CLIENT_APIS) -+ .unwrap() -+ .to_string_lossy(); -+ client_apis -+ .split(' ') -+ .any(|client_api| client_api == "OpenGL") -+ } else { -+ false -+ }; -+ egl.bind_api(if supports_opengl { -+ khronos_egl::OPENGL_API -+ } else { -+ khronos_egl::OPENGL_ES_API -+ }) -+ .unwrap(); -+ -+ let needs_robustness = true; -+ let mut khr_context_flags = 0; -+ let supports_khr_context = display_extensions.contains("EGL_KHR_create_context"); -+ -+ let mut context_attributes = vec![]; -+ let mut gl_context_attributes = vec![]; -+ let mut gles_context_attributes = vec![]; -+ gl_context_attributes.push(khronos_egl::CONTEXT_MAJOR_VERSION); -+ gl_context_attributes.push(3); -+ gl_context_attributes.push(khronos_egl::CONTEXT_MINOR_VERSION); -+ gl_context_attributes.push(3); -+ if supports_opengl && force_gles_minor_version != wgt::Gles3MinorVersion::Automatic { -+ log::warn!("Ignoring specified GLES minor version as OpenGL is used"); -+ } -+ gles_context_attributes.push(khronos_egl::CONTEXT_MAJOR_VERSION); -+ gles_context_attributes.push(3); // Request GLES 3.0 or higher -+ if force_gles_minor_version != wgt::Gles3MinorVersion::Automatic { -+ gles_context_attributes.push(khronos_egl::CONTEXT_MINOR_VERSION); -+ gles_context_attributes.push(match force_gles_minor_version { -+ wgt::Gles3MinorVersion::Automatic => unreachable!(), -+ wgt::Gles3MinorVersion::Version0 => 0, -+ wgt::Gles3MinorVersion::Version1 => 1, -+ wgt::Gles3MinorVersion::Version2 => 2, -+ }); -+ } -+ if flags.contains(wgt::InstanceFlags::DEBUG) { -+ if version >= (1, 5) { -+ log::debug!("\tEGL context: +debug"); -+ context_attributes.push(khronos_egl::CONTEXT_OPENGL_DEBUG); -+ context_attributes.push(khronos_egl::TRUE as _); -+ } else if supports_khr_context { -+ log::debug!("\tEGL context: +debug KHR"); -+ khr_context_flags |= EGL_CONTEXT_OPENGL_DEBUG_BIT_KHR; -+ } else { -+ log::debug!("\tEGL context: -debug"); -+ } -+ } -+ if needs_robustness { -+ //Note: the core version can fail if robustness is not supported -+ // (regardless of whether the extension is supported!). -+ // In fact, Angle does precisely that awful behavior, so we don't try it there. -+ if version >= (1, 5) && !display_extensions.contains("EGL_ANGLE_") { -+ log::debug!("\tEGL context: +robust access"); -+ context_attributes.push(khronos_egl::CONTEXT_OPENGL_ROBUST_ACCESS); -+ context_attributes.push(khronos_egl::TRUE as _); -+ } else if display_extensions.contains("EGL_EXT_create_context_robustness") { -+ log::debug!("\tEGL context: +robust access EXT"); -+ context_attributes.push(EGL_CONTEXT_OPENGL_ROBUST_ACCESS_EXT); -+ context_attributes.push(khronos_egl::TRUE as _); -+ } else { -+ //Note: we aren't trying `EGL_CONTEXT_OPENGL_ROBUST_ACCESS_BIT_KHR` -+ // because it's for desktop GL only, not GLES. -+ log::warn!("\tEGL context: -robust access"); -+ } -+ } -+ if khr_context_flags != 0 { -+ context_attributes.push(EGL_CONTEXT_FLAGS_KHR); -+ context_attributes.push(khr_context_flags); -+ } -+ context_attributes.push(khronos_egl::NONE); -+ -+ gl_context_attributes.extend(&context_attributes); -+ gles_context_attributes.extend(&context_attributes); -+ -+ // ALVR: share GL objects with the app's current EGL context, so textures created by the -+ // app (glGenTextures) are visible inside this context. -+ let share_context = egl.get_current_context(); -+ log::info!("ALVR GLES: share context = {:?}", share_context.map(|c| c.as_ptr())); -+ -+ let context = if supports_opengl { -+ egl.create_context(display, config, None, &gl_context_attributes) -+ .or_else(|_| { -+ egl.bind_api(khronos_egl::OPENGL_ES_API).unwrap(); -+ egl.create_context(display, config, None, &gles_context_attributes) -+ }) -+ .map_err(|e| { -+ crate::InstanceError::with_source( -+ String::from("unable to create OpenGL or GLES 3.x context"), -+ e, -+ ) -+ }) -+ } else { -+ egl.create_context(display, config, None, &gles_context_attributes) -+ .map_err(|e| { -+ crate::InstanceError::with_source( -+ String::from("unable to create GLES 3.x context"), -+ e, -+ ) -+ }) -+ }?; -+ -+ // Testing if context can be binded without surface -+ // and creating dummy pbuffer surface if not. -+ let pbuffer = if version >= (1, 5) -+ || display_extensions.contains("EGL_KHR_surfaceless_context") -+ || cfg!(Emscripten) -+ { -+ log::debug!("\tEGL context: +surfaceless"); -+ None -+ } else { -+ let attributes = [ -+ khronos_egl::WIDTH, -+ 1, -+ khronos_egl::HEIGHT, -+ 1, -+ khronos_egl::NONE, -+ ]; -+ egl.create_pbuffer_surface(display, config, &attributes) -+ .map(Some) -+ .map_err(|e| { -+ crate::InstanceError::with_source( -+ String::from("error in create_pbuffer_surface"), -+ e, -+ ) -+ })? -+ }; -+ -+ Ok(Self { -+ egl: EglContext { -+ instance: egl, -+ display, -+ raw: context, -+ pbuffer, -+ version, -+ }, -+ version, -+ supports_native_window, -+ config, -+ wl_display: None, -+ srgb_kind, -+ force_gles_minor_version, -+ }) -+ } -+} -+ -+impl Drop for Inner { -+ fn drop(&mut self) { -+ if let Err(e) = self -+ .egl -+ .instance -+ .destroy_context(self.egl.display, self.egl.raw) -+ { -+ log::warn!("Error in destroy_context: {:?}", e); -+ } -+ -+ if let Err(e) = terminate_display(&self.egl.instance, self.egl.display) { -+ log::warn!("Error in terminate: {:?}", e); -+ } -+ } -+} -+ -+#[derive(Clone, Copy, Debug, PartialEq)] -+enum WindowKind { -+ Wayland, -+ X11, -+ AngleX11, -+ Unknown, -+} -+ -+#[derive(Clone, Debug)] -+struct WindowSystemInterface { -+ display_owner: Option>, -+ kind: WindowKind, -+} -+ -+pub struct Instance { -+ wsi: WindowSystemInterface, -+ flags: wgt::InstanceFlags, -+ options: wgt::GlBackendOptions, -+ inner: Mutex, -+} -+ -+impl Instance { -+ pub fn raw_display(&self) -> khronos_egl::Display { -+ self.inner -+ .try_lock() -+ .expect("Could not lock instance. This is most-likely a deadlock.") -+ .egl -+ .display -+ } -+ -+ /// Returns the version of the EGL display. -+ pub fn egl_version(&self) -> (i32, i32) { -+ self.inner -+ .try_lock() -+ .expect("Could not lock instance. This is most-likely a deadlock.") -+ .version -+ } -+ -+ pub fn egl_config(&self) -> khronos_egl::Config { -+ self.inner -+ .try_lock() -+ .expect("Could not lock instance. This is most-likely a deadlock.") -+ .config -+ } -+} -+ -+unsafe impl Send for Instance {} -+unsafe impl Sync for Instance {} -+ -+impl crate::Instance for Instance { -+ type A = super::Api; -+ -+ unsafe fn init(desc: &crate::InstanceDescriptor) -> Result { -+ profiling::scope!("Init OpenGL (EGL) Backend"); -+ #[cfg(Emscripten)] -+ let egl_result: Result = -+ Ok(khronos_egl::Instance::new(khronos_egl::Static)); -+ -+ #[cfg(not(Emscripten))] -+ let egl_result = if cfg!(windows) { -+ unsafe { -+ khronos_egl::DynamicInstance::::load_required_from_filename( -+ "libEGL.dll", -+ ) -+ } -+ } else if cfg!(target_vendor = "apple") { -+ unsafe { -+ khronos_egl::DynamicInstance::::load_required_from_filename( -+ "libEGL.dylib", -+ ) -+ } -+ } else { -+ unsafe { khronos_egl::DynamicInstance::::load_required() } -+ }; -+ let egl = match egl_result { -+ Ok(egl) => Arc::new(egl), -+ Err(e) => { -+ return Err(crate::InstanceError::with_source( -+ String::from("unable to open libEGL"), -+ e, -+ )); -+ } -+ }; -+ -+ let client_extensions = egl.query_string(None, khronos_egl::EXTENSIONS); -+ -+ let client_ext_str = match client_extensions { -+ Ok(ext) => ext.to_string_lossy().into_owned(), -+ Err(_) => String::new(), -+ }; -+ log::debug!( -+ "Client extensions: {:#?}", -+ client_ext_str.split_whitespace().collect::>() -+ ); -+ -+ let wayland_library = if client_ext_str.contains("EGL_EXT_platform_wayland") { -+ test_wayland_display() -+ } else { -+ None -+ }; -+ let x11_display_library = if client_ext_str.contains("EGL_EXT_platform_x11") { -+ open_x_display() -+ } else { -+ None -+ }; -+ let angle_x11_display_library = if client_ext_str.contains("EGL_ANGLE_platform_angle") { -+ open_x_display() -+ } else { -+ None -+ }; -+ -+ #[cfg(not(Emscripten))] -+ let egl1_5 = egl.upcast::(); -+ -+ #[cfg(Emscripten)] -+ let egl1_5: Option<&Arc> = Some(&egl); -+ -+ let (display, display_owner, wsi_kind) = -+ if let (Some(library), Some(egl)) = (wayland_library, egl1_5) { -+ log::info!("Using Wayland platform"); -+ let display_attributes = [khronos_egl::ATTRIB_NONE]; -+ let display = unsafe { -+ egl.get_platform_display( -+ EGL_PLATFORM_WAYLAND_KHR, -+ khronos_egl::DEFAULT_DISPLAY, -+ &display_attributes, -+ ) -+ } -+ .unwrap(); -+ (display, Some(Rc::new(library)), WindowKind::Wayland) -+ } else if let (Some(display_owner), Some(egl)) = (x11_display_library, egl1_5) { -+ log::info!("Using X11 platform"); -+ let display_attributes = [khronos_egl::ATTRIB_NONE]; -+ let display = unsafe { -+ egl.get_platform_display( -+ EGL_PLATFORM_X11_KHR, -+ display_owner.display.as_ptr(), -+ &display_attributes, -+ ) -+ } -+ .unwrap(); -+ (display, Some(Rc::new(display_owner)), WindowKind::X11) -+ } else if let (Some(display_owner), Some(egl)) = (angle_x11_display_library, egl1_5) { -+ log::info!("Using Angle platform with X11"); -+ let display_attributes = [ -+ EGL_PLATFORM_ANGLE_NATIVE_PLATFORM_TYPE_ANGLE as khronos_egl::Attrib, -+ EGL_PLATFORM_X11_KHR as khronos_egl::Attrib, -+ EGL_PLATFORM_ANGLE_DEBUG_LAYERS_ENABLED as khronos_egl::Attrib, -+ usize::from(desc.flags.contains(wgt::InstanceFlags::VALIDATION)), -+ khronos_egl::ATTRIB_NONE, -+ ]; -+ let display = unsafe { -+ egl.get_platform_display( -+ EGL_PLATFORM_ANGLE_ANGLE, -+ display_owner.display.as_ptr(), -+ &display_attributes, -+ ) -+ } -+ .unwrap(); -+ (display, Some(Rc::new(display_owner)), WindowKind::AngleX11) -+ } else if client_ext_str.contains("EGL_MESA_platform_surfaceless") { -+ log::warn!("No windowing system present. Using surfaceless platform"); -+ #[allow(clippy::unnecessary_literal_unwrap)] // This is only a literal on Emscripten -+ let egl = egl1_5.expect("Failed to get EGL 1.5 for surfaceless"); -+ let display = unsafe { -+ egl.get_platform_display( -+ EGL_PLATFORM_SURFACELESS_MESA, -+ khronos_egl::DEFAULT_DISPLAY, -+ &[khronos_egl::ATTRIB_NONE], -+ ) -+ } -+ .unwrap(); -+ -+ (display, None, WindowKind::Unknown) -+ } else { -+ log::warn!("EGL_MESA_platform_surfaceless not available. Using default platform"); -+ let display = unsafe { egl.get_display(khronos_egl::DEFAULT_DISPLAY) }.unwrap(); -+ (display, None, WindowKind::Unknown) -+ }; -+ -+ if desc.flags.contains(wgt::InstanceFlags::VALIDATION) -+ && client_ext_str.contains("EGL_KHR_debug") -+ { -+ log::debug!("Enabling EGL debug output"); -+ let function: EglDebugMessageControlFun = { -+ let addr = egl.get_proc_address("eglDebugMessageControlKHR").unwrap(); -+ unsafe { std::mem::transmute(addr) } -+ }; -+ let attributes = [ -+ EGL_DEBUG_MSG_CRITICAL_KHR as khronos_egl::Attrib, -+ 1, -+ EGL_DEBUG_MSG_ERROR_KHR as khronos_egl::Attrib, -+ 1, -+ EGL_DEBUG_MSG_WARN_KHR as khronos_egl::Attrib, -+ 1, -+ EGL_DEBUG_MSG_INFO_KHR as khronos_egl::Attrib, -+ 1, -+ khronos_egl::ATTRIB_NONE, -+ ]; -+ unsafe { (function)(Some(egl_debug_proc), attributes.as_ptr()) }; -+ } -+ -+ let inner = Inner::create( -+ desc.flags, -+ egl, -+ display, -+ desc.backend_options.gl.gles_minor_version, -+ )?; -+ -+ Ok(Instance { -+ wsi: WindowSystemInterface { -+ display_owner, -+ kind: wsi_kind, -+ }, -+ flags: desc.flags, -+ options: desc.backend_options.gl.clone(), -+ inner: Mutex::new(inner), -+ }) -+ } -+ -+ #[cfg_attr(target_os = "macos", allow(unused, unused_mut, unreachable_code))] -+ unsafe fn create_surface( -+ &self, -+ display_handle: raw_window_handle::RawDisplayHandle, -+ window_handle: raw_window_handle::RawWindowHandle, -+ ) -> Result { -+ use raw_window_handle::RawWindowHandle as Rwh; -+ -+ #[cfg_attr(any(target_os = "android", Emscripten), allow(unused_mut))] -+ let mut inner = self.inner.lock(); -+ -+ match (window_handle, display_handle) { -+ (Rwh::Xlib(_), _) => {} -+ (Rwh::Xcb(_), _) => {} -+ (Rwh::Win32(_), _) => {} -+ (Rwh::AppKit(_), _) => {} -+ (Rwh::OhosNdk(_), _) => {} -+ #[cfg(target_os = "android")] -+ (Rwh::AndroidNdk(handle), _) => { -+ let format = inner -+ .egl -+ .instance -+ .get_config_attrib( -+ inner.egl.display, -+ inner.config, -+ khronos_egl::NATIVE_VISUAL_ID, -+ ) -+ .unwrap(); -+ -+ let ret = unsafe { -+ ndk_sys::ANativeWindow_setBuffersGeometry( -+ handle -+ .a_native_window -+ .as_ptr() -+ .cast::(), -+ 0, -+ 0, -+ format, -+ ) -+ }; -+ -+ if ret != 0 { -+ return Err(crate::InstanceError::new(format!( -+ "error {ret} returned from ANativeWindow_setBuffersGeometry", -+ ))); -+ } -+ } -+ #[cfg(not(Emscripten))] -+ (Rwh::Wayland(_), raw_window_handle::RawDisplayHandle::Wayland(display_handle)) => { -+ if inner -+ .wl_display -+ .map(|ptr| ptr != display_handle.display.as_ptr()) -+ .unwrap_or(true) -+ { -+ /* Wayland displays are not sharable between surfaces so if the -+ * surface we receive from this handle is from a different -+ * display, we must re-initialize the context. -+ * -+ * See gfx-rs/gfx#3545 -+ */ -+ log::warn!("Re-initializing Gles context due to Wayland window"); -+ -+ use std::ops::DerefMut; -+ let display_attributes = [khronos_egl::ATTRIB_NONE]; -+ -+ let display = unsafe { -+ inner -+ .egl -+ .instance -+ .upcast::() -+ .unwrap() -+ .get_platform_display( -+ EGL_PLATFORM_WAYLAND_KHR, -+ display_handle.display.as_ptr(), -+ &display_attributes, -+ ) -+ } -+ .unwrap(); -+ -+ let new_inner = Inner::create( -+ self.flags, -+ Arc::clone(&inner.egl.instance), -+ display, -+ inner.force_gles_minor_version, -+ )?; -+ -+ let old_inner = std::mem::replace(inner.deref_mut(), new_inner); -+ inner.wl_display = Some(display_handle.display.as_ptr()); -+ -+ drop(old_inner); -+ } -+ } -+ #[cfg(Emscripten)] -+ (Rwh::Web(_), _) => {} -+ other => { -+ return Err(crate::InstanceError::new(format!( -+ "unsupported window: {other:?}" -+ ))); -+ } -+ }; -+ -+ inner.egl.unmake_current(); -+ -+ Ok(Surface { -+ egl: inner.egl.clone(), -+ wsi: self.wsi.clone(), -+ config: inner.config, -+ presentable: inner.supports_native_window, -+ raw_window_handle: window_handle, -+ swapchain: RwLock::new(None), -+ srgb_kind: inner.srgb_kind, -+ }) -+ } -+ -+ unsafe fn enumerate_adapters( -+ &self, -+ _surface_hint: Option<&Surface>, -+ ) -> Vec> { -+ let inner = self.inner.lock(); -+ inner.egl.make_current(); -+ -+ let mut gl = unsafe { -+ glow::Context::from_loader_function(|name| { -+ inner -+ .egl -+ .instance -+ .get_proc_address(name) -+ .map_or(ptr::null(), |p| p as *const _) -+ }) -+ }; -+ -+ // In contrast to OpenGL ES, OpenGL requires explicitly enabling sRGB conversions, -+ // as otherwise the user has to do the sRGB conversion. -+ if !matches!(inner.srgb_kind, SrgbFrameBufferKind::None) { -+ unsafe { gl.enable(glow::FRAMEBUFFER_SRGB) }; -+ } -+ -+ if self.flags.contains(wgt::InstanceFlags::DEBUG) && gl.supports_debug() { -+ log::debug!("Max label length: {}", unsafe { -+ gl.get_parameter_i32(glow::MAX_LABEL_LENGTH) -+ }); -+ } -+ -+ if self.flags.contains(wgt::InstanceFlags::VALIDATION) && gl.supports_debug() { -+ log::debug!("Enabling GLES debug output"); -+ unsafe { gl.enable(glow::DEBUG_OUTPUT) }; -+ unsafe { gl.debug_message_callback(super::gl_debug_message_callback) }; -+ } -+ -+ // Wrap in ManuallyDrop to make it easier to "current" the GL context before dropping this -+ // GLOW context, which could also happen if a panic occurs after we uncurrent the context -+ // below but before AdapterContext is constructed. -+ let gl = ManuallyDrop::new(gl); -+ inner.egl.unmake_current(); -+ -+ unsafe { -+ super::Adapter::expose( -+ AdapterContext { -+ glow: Mutex::new(gl), -+ egl: Some(inner.egl.clone()), -+ }, -+ self.options.clone(), -+ ) -+ } -+ .into_iter() -+ .collect() -+ } -+} -+ -+impl super::Adapter { -+ /// Creates a new external adapter using the specified loader function. -+ /// -+ /// # Safety -+ /// -+ /// - The underlying OpenGL ES context must be current. -+ /// - The underlying OpenGL ES context must be current when interfacing with any objects returned by -+ /// wgpu-hal from this adapter. -+ /// - The underlying OpenGL ES context must be current when dropping this adapter and when -+ /// dropping any objects returned from this adapter. -+ pub unsafe fn new_external( -+ fun: impl FnMut(&str) -> *const ffi::c_void, -+ options: wgt::GlBackendOptions, -+ ) -> Option> { -+ let context = unsafe { glow::Context::from_loader_function(fun) }; -+ unsafe { -+ Self::expose( -+ AdapterContext { -+ glow: Mutex::new(ManuallyDrop::new(context)), -+ egl: None, -+ }, -+ options, -+ ) -+ } -+ } -+ -+ pub fn adapter_context(&self) -> &AdapterContext { -+ &self.shared.context -+ } -+} -+ -+impl super::Device { -+ /// Returns the underlying EGL context. -+ pub fn context(&self) -> &AdapterContext { -+ &self.shared.context -+ } -+} -+ -+#[derive(Debug)] -+pub struct Swapchain { -+ surface: khronos_egl::Surface, -+ wl_window: Option<*mut raw::c_void>, -+ framebuffer: glow::Framebuffer, -+ renderbuffer: glow::Renderbuffer, -+ /// Extent because the window lies -+ extent: wgt::Extent3d, -+ format: wgt::TextureFormat, -+ format_desc: super::TextureFormatDesc, -+ #[allow(unused)] -+ sample_type: wgt::TextureSampleType, -+} -+ -+#[derive(Debug)] -+pub struct Surface { -+ egl: EglContext, -+ wsi: WindowSystemInterface, -+ config: khronos_egl::Config, -+ pub(super) presentable: bool, -+ raw_window_handle: raw_window_handle::RawWindowHandle, -+ swapchain: RwLock>, -+ srgb_kind: SrgbFrameBufferKind, -+} -+ -+unsafe impl Send for Surface {} -+unsafe impl Sync for Surface {} -+ -+impl Surface { -+ pub(super) unsafe fn present( -+ &self, -+ _suf_texture: super::Texture, -+ context: &AdapterContext, -+ ) -> Result<(), crate::SurfaceError> { -+ let gl = unsafe { context.get_without_egl_lock() }; -+ let swapchain = self.swapchain.read(); -+ let sc = swapchain.as_ref().unwrap(); -+ -+ self.egl -+ .instance -+ .make_current( -+ self.egl.display, -+ Some(sc.surface), -+ Some(sc.surface), -+ Some(self.egl.raw), -+ ) -+ .map_err(|e| { -+ log::error!("make_current(surface) failed: {}", e); -+ crate::SurfaceError::Lost -+ })?; -+ -+ unsafe { gl.disable(glow::SCISSOR_TEST) }; -+ unsafe { gl.color_mask(true, true, true, true) }; -+ -+ unsafe { gl.bind_framebuffer(glow::DRAW_FRAMEBUFFER, None) }; -+ unsafe { gl.bind_framebuffer(glow::READ_FRAMEBUFFER, Some(sc.framebuffer)) }; -+ -+ if !matches!(self.srgb_kind, SrgbFrameBufferKind::None) { -+ // Disable sRGB conversions for `glBlitFramebuffer` as behavior does diverge between -+ // drivers and formats otherwise and we want to ensure no sRGB conversions happen. -+ unsafe { gl.disable(glow::FRAMEBUFFER_SRGB) }; -+ } -+ -+ // Note the Y-flipping here. GL's presentation is not flipped, -+ // but main rendering is. Therefore, we Y-flip the output positions -+ // in the shader, and also this blit. -+ unsafe { -+ gl.blit_framebuffer( -+ 0, -+ sc.extent.height as i32, -+ sc.extent.width as i32, -+ 0, -+ 0, -+ 0, -+ sc.extent.width as i32, -+ sc.extent.height as i32, -+ glow::COLOR_BUFFER_BIT, -+ glow::NEAREST, -+ ) -+ }; -+ -+ if !matches!(self.srgb_kind, SrgbFrameBufferKind::None) { -+ unsafe { gl.enable(glow::FRAMEBUFFER_SRGB) }; -+ } -+ -+ unsafe { gl.bind_framebuffer(glow::READ_FRAMEBUFFER, None) }; -+ -+ self.egl -+ .instance -+ .swap_buffers(self.egl.display, sc.surface) -+ .map_err(|e| { -+ log::error!("swap_buffers failed: {}", e); -+ crate::SurfaceError::Lost -+ // TODO: should we unset the current context here? -+ })?; -+ self.egl -+ .instance -+ .make_current(self.egl.display, None, None, None) -+ .map_err(|e| { -+ log::error!("make_current(null) failed: {}", e); -+ crate::SurfaceError::Lost -+ })?; -+ -+ Ok(()) -+ } -+ -+ unsafe fn unconfigure_impl( -+ &self, -+ device: &super::Device, -+ ) -> Option<(khronos_egl::Surface, Option<*mut raw::c_void>)> { -+ let gl = &device.shared.context.lock(); -+ match self.swapchain.write().take() { -+ Some(sc) => { -+ unsafe { gl.delete_renderbuffer(sc.renderbuffer) }; -+ unsafe { gl.delete_framebuffer(sc.framebuffer) }; -+ Some((sc.surface, sc.wl_window)) -+ } -+ None => None, -+ } -+ } -+ -+ pub fn supports_srgb(&self) -> bool { -+ match self.srgb_kind { -+ SrgbFrameBufferKind::None => false, -+ _ => true, -+ } -+ } -+} -+ -+impl crate::Surface for Surface { -+ type A = super::Api; -+ -+ unsafe fn configure( -+ &self, -+ device: &super::Device, -+ config: &crate::SurfaceConfiguration, -+ ) -> Result<(), crate::SurfaceError> { -+ use raw_window_handle::RawWindowHandle as Rwh; -+ -+ let (surface, wl_window) = match unsafe { self.unconfigure_impl(device) } { -+ Some(pair) => pair, -+ None => { -+ let mut wl_window = None; -+ let (mut temp_xlib_handle, mut temp_xcb_handle); -+ let native_window_ptr = match (self.wsi.kind, self.raw_window_handle) { -+ (WindowKind::Unknown | WindowKind::X11, Rwh::Xlib(handle)) => { -+ temp_xlib_handle = handle.window; -+ ptr::from_mut(&mut temp_xlib_handle).cast::() -+ } -+ (WindowKind::AngleX11, Rwh::Xlib(handle)) => handle.window as *mut ffi::c_void, -+ (WindowKind::Unknown | WindowKind::X11, Rwh::Xcb(handle)) => { -+ temp_xcb_handle = handle.window; -+ ptr::from_mut(&mut temp_xcb_handle).cast::() -+ } -+ (WindowKind::AngleX11, Rwh::Xcb(handle)) => { -+ handle.window.get() as *mut ffi::c_void -+ } -+ (WindowKind::Unknown, Rwh::AndroidNdk(handle)) => { -+ handle.a_native_window.as_ptr() -+ } -+ (WindowKind::Unknown, Rwh::OhosNdk(handle)) => handle.native_window.as_ptr(), -+ (WindowKind::Wayland, Rwh::Wayland(handle)) => { -+ let library = &self.wsi.display_owner.as_ref().unwrap().library; -+ let wl_egl_window_create: libloading::Symbol = -+ unsafe { library.get(c"wl_egl_window_create".to_bytes()) }.unwrap(); -+ let window = -+ unsafe { wl_egl_window_create(handle.surface.as_ptr(), 640, 480) } -+ .cast(); -+ wl_window = Some(window); -+ window -+ } -+ #[cfg(Emscripten)] -+ (WindowKind::Unknown, Rwh::Web(handle)) => handle.id as *mut ffi::c_void, -+ (WindowKind::Unknown, Rwh::Win32(handle)) => { -+ handle.hwnd.get() as *mut ffi::c_void -+ } -+ (WindowKind::Unknown, Rwh::AppKit(handle)) => { -+ #[cfg(not(target_os = "macos"))] -+ let window_ptr = handle.ns_view.as_ptr(); -+ #[cfg(target_os = "macos")] -+ let window_ptr = { -+ use objc::{msg_send, runtime::Object, sel, sel_impl}; -+ // ns_view always have a layer and don't need to verify that it exists. -+ let layer: *mut Object = -+ msg_send![handle.ns_view.as_ptr().cast::(), layer]; -+ layer.cast::() -+ }; -+ window_ptr -+ } -+ _ => { -+ log::warn!( -+ "Initialized platform {:?} doesn't work with window {:?}", -+ self.wsi.kind, -+ self.raw_window_handle -+ ); -+ return Err(crate::SurfaceError::Other("incompatible window kind")); -+ } -+ }; -+ -+ let mut attributes = vec![ -+ khronos_egl::RENDER_BUFFER, -+ // We don't want any of the buffering done by the driver, because we -+ // manage a swapchain on our side. -+ // Some drivers just fail on surface creation seeing `EGL_SINGLE_BUFFER`. -+ if cfg!(any( -+ target_os = "android", -+ target_os = "macos", -+ target_env = "ohos" -+ )) || cfg!(windows) -+ || self.wsi.kind == WindowKind::AngleX11 -+ { -+ khronos_egl::BACK_BUFFER -+ } else { -+ khronos_egl::SINGLE_BUFFER -+ }, -+ ]; -+ if config.format.is_srgb() { -+ match self.srgb_kind { -+ SrgbFrameBufferKind::None => {} -+ SrgbFrameBufferKind::Core => { -+ attributes.push(khronos_egl::GL_COLORSPACE); -+ attributes.push(khronos_egl::GL_COLORSPACE_SRGB); -+ } -+ SrgbFrameBufferKind::Khr => { -+ attributes.push(EGL_GL_COLORSPACE_KHR as i32); -+ attributes.push(EGL_GL_COLORSPACE_SRGB_KHR as i32); -+ } -+ } -+ } -+ attributes.push(khronos_egl::ATTRIB_NONE as i32); -+ -+ #[cfg(not(Emscripten))] -+ let egl1_5 = self.egl.instance.upcast::(); -+ -+ #[cfg(Emscripten)] -+ let egl1_5: Option<&Arc> = Some(&self.egl.instance); -+ -+ // Careful, we can still be in 1.4 version even if `upcast` succeeds -+ let raw_result = match egl1_5 { -+ Some(egl) if self.wsi.kind != WindowKind::Unknown => { -+ let attributes_usize = attributes -+ .into_iter() -+ .map(|v| v as usize) -+ .collect::>(); -+ unsafe { -+ egl.create_platform_window_surface( -+ self.egl.display, -+ self.config, -+ native_window_ptr, -+ &attributes_usize, -+ ) -+ } -+ } -+ _ => unsafe { -+ self.egl.instance.create_window_surface( -+ self.egl.display, -+ self.config, -+ native_window_ptr, -+ Some(&attributes), -+ ) -+ }, -+ }; -+ -+ match raw_result { -+ Ok(raw) => (raw, wl_window), -+ Err(e) => { -+ log::warn!("Error in create_window_surface: {:?}", e); -+ return Err(crate::SurfaceError::Lost); -+ } -+ } -+ } -+ }; -+ -+ if let Some(window) = wl_window { -+ let library = &self.wsi.display_owner.as_ref().unwrap().library; -+ let wl_egl_window_resize: libloading::Symbol = -+ unsafe { library.get(c"wl_egl_window_resize".to_bytes()) }.unwrap(); -+ unsafe { -+ wl_egl_window_resize( -+ window, -+ config.extent.width as i32, -+ config.extent.height as i32, -+ 0, -+ 0, -+ ) -+ }; -+ } -+ -+ let format_desc = device.shared.describe_texture_format(config.format); -+ let gl = &device.shared.context.lock(); -+ let renderbuffer = unsafe { gl.create_renderbuffer() }.map_err(|error| { -+ log::error!("Internal swapchain renderbuffer creation failed: {error}"); -+ crate::DeviceError::OutOfMemory -+ })?; -+ unsafe { gl.bind_renderbuffer(glow::RENDERBUFFER, Some(renderbuffer)) }; -+ unsafe { -+ gl.renderbuffer_storage( -+ glow::RENDERBUFFER, -+ format_desc.internal, -+ config.extent.width as _, -+ config.extent.height as _, -+ ) -+ }; -+ let framebuffer = unsafe { gl.create_framebuffer() }.map_err(|error| { -+ log::error!("Internal swapchain framebuffer creation failed: {error}"); -+ crate::DeviceError::OutOfMemory -+ })?; -+ unsafe { gl.bind_framebuffer(glow::READ_FRAMEBUFFER, Some(framebuffer)) }; -+ unsafe { -+ gl.framebuffer_renderbuffer( -+ glow::READ_FRAMEBUFFER, -+ glow::COLOR_ATTACHMENT0, -+ glow::RENDERBUFFER, -+ Some(renderbuffer), -+ ) -+ }; -+ unsafe { gl.bind_renderbuffer(glow::RENDERBUFFER, None) }; -+ unsafe { gl.bind_framebuffer(glow::READ_FRAMEBUFFER, None) }; -+ -+ let mut swapchain = self.swapchain.write(); -+ *swapchain = Some(Swapchain { -+ surface, -+ wl_window, -+ renderbuffer, -+ framebuffer, -+ extent: config.extent, -+ format: config.format, -+ format_desc, -+ sample_type: wgt::TextureSampleType::Float { filterable: false }, -+ }); -+ -+ Ok(()) -+ } -+ -+ unsafe fn unconfigure(&self, device: &super::Device) { -+ if let Some((surface, wl_window)) = unsafe { self.unconfigure_impl(device) } { -+ self.egl -+ .instance -+ .destroy_surface(self.egl.display, surface) -+ .unwrap(); -+ if let Some(window) = wl_window { -+ let library = &self -+ .wsi -+ .display_owner -+ .as_ref() -+ .expect("unsupported window") -+ .library; -+ let wl_egl_window_destroy: libloading::Symbol = -+ unsafe { library.get(c"wl_egl_window_destroy".to_bytes()) }.unwrap(); -+ unsafe { wl_egl_window_destroy(window) }; -+ } -+ } -+ } -+ -+ unsafe fn acquire_texture( -+ &self, -+ _timeout_ms: Option, //TODO -+ _fence: &super::Fence, -+ ) -> Result>, crate::SurfaceError> { -+ let swapchain = self.swapchain.read(); -+ let sc = swapchain.as_ref().unwrap(); -+ let texture = super::Texture { -+ inner: super::TextureInner::Renderbuffer { -+ raw: sc.renderbuffer, -+ }, -+ drop_guard: None, -+ array_layer_count: 1, -+ mip_level_count: 1, -+ format: sc.format, -+ format_desc: sc.format_desc.clone(), -+ copy_size: crate::CopyExtent { -+ width: sc.extent.width, -+ height: sc.extent.height, -+ depth: 1, -+ }, -+ }; -+ Ok(Some(crate::AcquiredSurfaceTexture { -+ texture, -+ suboptimal: false, -+ })) -+ } -+ unsafe fn discard_texture(&self, _texture: super::Texture) {} -+} -diff -ruN a/src/gles/queue.rs b/src/gles/queue.rs ---- a/src/gles/queue.rs 2026-10-10 19:59:35.659717497 +0900 -+++ b/src/gles/queue.rs 2026-10-10 19:59:35.673313980 +0900 -@@ -144,7 +144,14 @@ - get_2d_target(target, view.array_layers.start), - Some(raw), - view.mip_levels.start as i32, -- ) -+ ); -+ log::info!( -+ "ALVR GLES ATTACH: texture={} attachment=0x{:X} error=0x{:X} framebuffer=0x{:X}", -+ raw.0, -+ attachment, -+ gl.get_error(), -+ gl.check_framebuffer_status(glow::DRAW_FRAMEBUFFER), -+ ); - }; - } - } -@@ -1818,8 +1825,17 @@ - _surface_textures: &[&super::Texture], - (signal_fence, signal_value): (&mut super::Fence, crate::FenceValue), - ) -> Result<(), crate::DeviceError> { -+ log::info!( -+ "ALVR GLES SUBMIT: entered submit(), command_buffers={}", -+ command_buffers.len() -+ ); -+ - let shared = Arc::clone(&self.shared); -+ -+ log::info!("ALVR GLES SUBMIT: about to lock EGL context"); - let gl = &shared.context.lock(); -+ log::info!("ALVR GLES SUBMIT: EGL context locked"); -+ - for cmd_buf in command_buffers.iter() { - // The command encoder assumes a default state when encoding the command buffer. - // Always reset the state between command_buffers to reflect this assumption. Do From 551f605c78604685737224507485ac563d9e819e Mon Sep 17 00:00:00 2001 From: Rob Oates Date: Wed, 14 Oct 2026 21:48:49 +0900 Subject: [PATCH 20/29] Reduce display prediction interval to 30 ms to stop reprojection overshoot --- code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp index f88cff06..933c833f 100644 --- a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp +++ b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp @@ -84,7 +84,7 @@ uint64_t HEAD_ID = alvr_path_string_to_id("/user/head"); const uint64_t ESTIMATED_PREDICTION_OFFSET_NS = 30e6; // Note: the Cardboard SDK cannot estimate display time and an heuristic is used instead. -const uint64_t VSYNC_QUEUE_INTERVAL_NS = 50e6; +const uint64_t VSYNC_QUEUE_INTERVAL_NS = 30e6; const float FLOOR_HEIGHT = 1.5; const int MAXIMUM_TRACKING_FRAMES = 360; From 05054b7bbdcc21ce3021b90da1662ee4a6701849 Mon Sep 17 00:00:00 2001 From: Rob Oates Date: Mon, 19 Oct 2026 00:33:28 +0900 Subject: [PATCH 21/29] Add client-side upscaling option and stream FOV scale --- .../PhoneVR/app/src/main/cpp/alvr_main.cpp | 82 +++++++++++++++++-- 1 file changed, 73 insertions(+), 9 deletions(-) diff --git a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp index 933c833f..a8f69958 100644 --- a/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp +++ b/code/mobile/android/PhoneVR/app/src/main/cpp/alvr_main.cpp @@ -84,7 +84,29 @@ uint64_t HEAD_ID = alvr_path_string_to_id("/user/head"); const uint64_t ESTIMATED_PREDICTION_OFFSET_NS = 30e6; // Note: the Cardboard SDK cannot estimate display time and an heuristic is used instead. +// Tested on the Oppo Reno 9 A: 50 ms made the streamed picture overshoot when a fast head turn +// stopped (felt over-sensitive); 30 ms removes that. Candidate for per-phone calibration. const uint64_t VSYNC_QUEUE_INTERVAL_NS = 30e6; + +// Client-side upscaling (ALVR's Snapdragon Game Super Resolution pass). The server streams a +// smaller picture, the decoder works on that, and the client scales it up to (about) the phone's +// real per-eye size. Only worth turning on when the stream is smaller than the display, e.g. to +// save Wi-Fi bandwidth or decoder load: set the server's video resolution lower in the dashboard. +// The upscale factor is worked out from the stream size, so there is nothing else to tune. +constexpr bool ENABLE_UPSCALING = true; +constexpr float UPSCALING_MAX_FACTOR = 3.0f; // the same limit as ALVR's own slider +// Same defaults as ALVR's own settings. +constexpr bool UPSCALING_EDGE_DIRECTION = true; +constexpr float UPSCALING_EDGE_THRESHOLD = 4.0f; +constexpr float UPSCALING_EDGE_SHARPNESS = 2.0f; + +// Zoom of the streamed picture. The server renders with the field of view we report; reporting a +// narrower one than the Cardboard lens really shows gives a bigger, closer-looking picture with +// more pixels per degree (so text is easier to read). 1.0 = exactly what the lens geometry says. +// The value scales the tangent of the half-angles, so 0.8 = about 25% zoom. The lobby is not +// affected. Too small a value makes straight lines bend slightly toward the edges. +// Tested on the Oppo Reno 9 A against the native client, which looked about 25% more zoomed in. +constexpr float STREAM_FOV_SCALE = 0.8f; const float FLOOR_HEIGHT = 1.5; const int MAXIMUM_TRACKING_FRAMES = 360; @@ -130,7 +152,7 @@ constexpr int ARCORE_WORLD_UP_AXIS = 1; // 2 = like 1, but until a floor is found it assumes the headset started // ARCORE_ASSUMED_START_HEIGHT_M above the floor, and it eases toward the measured floor // height instead of jumping. -constexpr int ARCORE_HEIGHT_MODE = 0; +constexpr int ARCORE_HEIGHT_MODE = 2; // Mode 2 only: assumed height above the floor at the start, before any floor has been detected. constexpr float ARCORE_ASSUMED_START_HEIGHT_M = 1.5f; // Mode 2 only: how slowly the height eases toward the measured floor. Larger = slower. @@ -261,6 +283,17 @@ AlvrFov getFov(CardboardEye eye) { return fov; } +// The field of view reported to the server, i.e. the one the stream is rendered with. The lobby +// keeps using getFov() directly. See STREAM_FOV_SCALE. +AlvrFov getStreamFov(CardboardEye eye) { + AlvrFov fov = getFov(eye); + fov.left = atanf(tanf(fov.left) * STREAM_FOV_SCALE); + fov.right = atanf(tanf(fov.right) * STREAM_FOV_SCALE); + fov.up = atanf(tanf(fov.up) * STREAM_FOV_SCALE); + fov.down = atanf(tanf(fov.down) * STREAM_FOV_SCALE); + return fov; +} + // Quaternion helpers for the orientation correction (same maths as the v20 branch). static AlvrQuat quatMultiply(const AlvrQuat &a, const AlvrQuat &b) { return {a.w * b.x + a.x * b.w + a.y * b.z - a.z * b.y, @@ -921,8 +954,8 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render CTX.eyeOffsets[eye] = matrix[12]; } - CTX.fovArr[kLeft] = getFov(kLeft); - CTX.fovArr[kRight] = getFov(kRight); + CTX.fovArr[kLeft] = getStreamFov(kLeft); + CTX.fovArr[kRight] = getStreamFov(kRight); info("renderingParamsChanged, sending new view configs (FOV) to alvr"); sendViewParams(); @@ -1057,6 +1090,30 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render CTX.lastStreamBuffer = nullptr; CTX.videoFramesSeen = 0; + // Upscaling: the stream is started_view_width x height, but when upscaling is on ALVR + // renders into textures that are upscale_factor times bigger (it truncates the + // product to whole pixels, exactly as done here), so the textures we hand over have + // to be that big too. A texture of the wrong size is what turned the screen black. + float upscaleFactor = 1.0f; + if (ENABLE_UPSCALING) { + float wantW = (float) (CTX.screenWidth / 2) / (float) started.view_width; + float wantH = (float) CTX.screenHeight / (float) started.view_height; + float want = std::min(wantW, wantH); + if (want >= 1.05f) { // stream is clearly smaller than the display + upscaleFactor = std::min(want, UPSCALING_MAX_FACTOR); + } + } + const bool upscaling = upscaleFactor > 1.0f; + const uint32_t targetWidth = (uint32_t) ((float) started.view_width * upscaleFactor); + const uint32_t targetHeight = (uint32_t) ((float) started.view_height * upscaleFactor); + info("Upscaling %s: stream %dx%d, factor %.3f, textures %ux%u", + upscaling ? "ON" : "off", + started.view_width, + started.view_height, + upscaleFactor, + targetWidth, + targetHeight); + GL(glGenTextures(2, CTX.streamTextures)); for (auto &streamTexture : CTX.streamTextures) { @@ -1068,16 +1125,16 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render GL(glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, - started.view_width, - started.view_height, + targetWidth, + targetHeight, 0, GL_RGB, GL_UNSIGNED_BYTE, nullptr)); } - CTX.fovArr[0] = getFov((CardboardEye) 0); - CTX.fovArr[1] = getFov((CardboardEye) 1); + CTX.fovArr[0] = getStreamFov((CardboardEye) 0); + CTX.fovArr[1] = getStreamFov((CardboardEye) 1); sendViewParams(); info("ALVR Poll Event: ALVR_EVENT_STREAMING_STARTED, View configs updated..."); @@ -1095,8 +1152,15 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render render_config.enable_foveation = started.enable_foveated_encoding; render_config.foveated_encoding = started.foveated_encoding; - // Not opting into server-side upscaling yet. - render_config.enable_upscaling = false; + // view_resolution above stays the size of the decoded stream; the textures are the + // upscaled size (see above). + render_config.enable_upscaling = upscaling; + if (upscaling) { + render_config.upscaling_edge_direction = UPSCALING_EDGE_DIRECTION; + render_config.upscaling_edge_threshold = UPSCALING_EDGE_THRESHOLD; + render_config.upscaling_edge_sharpness = UPSCALING_EDGE_SHARPNESS; + render_config.upscale_factor = upscaleFactor; + } ALVR_GL_CALL(alvr_start_stream_opengl(render_config)); From 3f3fab7f31fa9bd2e99500c8e67fde4394e526e1 Mon Sep 17 00:00:00 2001 From: Rob Oates Date: Tue, 20 Oct 2026 19:31:30 +0900 Subject: [PATCH 22/29] Add settings screen (ARCore switch, FOV zoom, floor mode); client upscaling option; 30 ms prediction; ARCore optional meta-data --- .../PhoneVR/app/src/main/AndroidManifest.xml | 8 ++ .../PhoneVR/app/src/main/cpp/alvr_main.cpp | 36 ++++-- .../viritualisres/phonevr/ALVRActivity.java | 20 +++- .../viritualisres/phonevr/InitActivity.kt | 4 + .../phonevr/SettingsActivity.java | 103 ++++++++++++++++++ .../app/src/main/res/layout/activity_init.xml | 11 ++ .../main/res/layout/activity_init_nogvr.xml | 11 ++ .../src/main/res/layout/activity_settings.xml | 103 ++++++++++++++++++ .../app/src/main/res/values/strings.xml | 14 +++ 9 files changed, 301 insertions(+), 9 deletions(-) create mode 100644 code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/SettingsActivity.java create mode 100644 code/mobile/android/PhoneVR/app/src/main/res/layout/activity_settings.xml diff --git a/code/mobile/android/PhoneVR/app/src/main/AndroidManifest.xml b/code/mobile/android/PhoneVR/app/src/main/AndroidManifest.xml index cec5caaf..528542fc 100644 --- a/code/mobile/android/PhoneVR/app/src/main/AndroidManifest.xml +++ b/code/mobile/android/PhoneVR/app/src/main/AndroidManifest.xml @@ -47,6 +47,11 @@ tools:replace="android:extractNativeLibs" android:extractNativeLibs="true"> + + + + 0 && blendDtNs < ARCORE_MAX_CORRECTION_GAP_NS) { @@ -815,6 +820,21 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_initia } } +// Settings from the app's Settings screen. Called right after initializeNative(), before the +// stream starts. +extern "C" JNIEXPORT void JNICALL +Java_viritualisres_phonevr_ALVRActivity_setStreamFovScaleNative(JNIEnv *, jobject, jfloat scale) { + // Keep it in a sane range whatever was stored. + CTX.streamFovScale = std::min(1.0f, std::max(0.5f, (float) scale)); + info("Setting: stream FOV scale %.2f", CTX.streamFovScale); +} + +extern "C" JNIEXPORT void JNICALL +Java_viritualisres_phonevr_ALVRActivity_setFloorModeNative(JNIEnv *, jobject, jint mode) { + CTX.heightMode = (mode >= 0 && mode <= 2) ? (int) mode : ARCORE_HEIGHT_MODE; + info("Setting: ARCore floor/height mode %d", CTX.heightMode); +} + extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_destroyNative(JNIEnv *, jobject) { alvr_destroy_opengl(); diff --git a/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/ALVRActivity.java b/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/ALVRActivity.java index 1b6cf0a4..365f93c8 100644 --- a/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/ALVRActivity.java +++ b/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/ALVRActivity.java @@ -105,7 +105,21 @@ public void onCreate(Bundle savedInstance) { float refreshRate = display.getRefreshRate(); Log.i(TAG, "Refresh rate: " + refreshRate); - initializeNative(width, height, refreshRate, true); + // Options chosen on the Settings screen (InitActivity -> Settings). + SharedPreferences appSettings = getSharedPreferences(SettingsActivity.PREFS, MODE_PRIVATE); + boolean enableARCore = + appSettings.getBoolean( + SettingsActivity.KEY_ARCORE_ENABLED, + SettingsActivity.DEFAULT_ARCORE_ENABLED); + Log.i(TAG, "ARCore enabled in settings: " + enableARCore); + + initializeNative(width, height, refreshRate, enableARCore); + setStreamFovScaleNative( + appSettings.getFloat( + SettingsActivity.KEY_FOV_SCALE, SettingsActivity.DEFAULT_FOV_SCALE)); + setFloorModeNative( + appSettings.getInt( + SettingsActivity.KEY_FLOOR_MODE, SettingsActivity.DEFAULT_FLOOR_MODE)); setContentView(R.layout.activity_vr); glView = findViewById(R.id.surface_view); @@ -287,6 +301,10 @@ private void setImmersiveSticky() { private native void initializeNative( int screenWidth, int screenHeight, float screenRefreshRate, boolean enableARCore); + private native void setStreamFovScaleNative(float scale); + + private native void setFloorModeNative(int mode); + private native void destroyNative(); private native void resumeNative(); diff --git a/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/InitActivity.kt b/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/InitActivity.kt index 1e951ca7..a0a5b57c 100644 --- a/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/InitActivity.kt +++ b/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/InitActivity.kt @@ -68,6 +68,10 @@ class InitActivity : AppCompatActivity() { } } + fun btOnClickSettings(view: View) { + startActivity(Intent(this, SettingsActivity::class.java)) + } + fun btOnClickALVRStreamer(view: View) { val prefs = getSharedPreferences("prefs", MODE_PRIVATE) // Save choice after first open. alvr_server as boolean diff --git a/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/SettingsActivity.java b/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/SettingsActivity.java new file mode 100644 index 00000000..ee0062a2 --- /dev/null +++ b/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/SettingsActivity.java @@ -0,0 +1,103 @@ +/* (C)2026 */ +package viritualisres.phonevr; + +import android.content.SharedPreferences; +import android.os.Bundle; +import android.view.View; +import android.widget.AdapterView; +import android.widget.ArrayAdapter; +import android.widget.SeekBar; +import android.widget.Spinner; +import android.widget.TextView; +import androidx.appcompat.app.AppCompatActivity; +import androidx.appcompat.widget.SwitchCompat; + +/** + * The settings screen shown from the start screen, before VR starts. Every change is saved at once + * to the "settings" preferences and read by ALVRActivity when VR starts. + */ +public class SettingsActivity extends AppCompatActivity { + + // The same file ALVRActivity already uses for max_brightness. + static final String PREFS = "settings"; + + static final String KEY_ARCORE_ENABLED = "arcore_enabled"; + static final String KEY_FOV_SCALE = "fov_scale"; + static final String KEY_FLOOR_MODE = "floor_mode"; + + static final boolean DEFAULT_ARCORE_ENABLED = true; + static final float DEFAULT_FOV_SCALE = 0.8f; + static final int DEFAULT_FLOOR_MODE = 2; + + // The slider covers 60% .. 100% in 1% steps (progress 0 .. 40). + private static final int FOV_MIN_PERCENT = 60; + private static final int FOV_MAX_PERCENT = 100; + + private SharedPreferences prefs; + private TextView fovValue; + + @Override + protected void onCreate(Bundle savedInstanceState) { + super.onCreate(savedInstanceState); + setContentView(R.layout.activity_settings); + prefs = getSharedPreferences(PREFS, MODE_PRIVATE); + + // ARCore on/off + SwitchCompat arcoreSwitch = findViewById(R.id.settings_arcore_switch); + arcoreSwitch.setChecked(prefs.getBoolean(KEY_ARCORE_ENABLED, DEFAULT_ARCORE_ENABLED)); + arcoreSwitch.setOnCheckedChangeListener( + (button, isChecked) -> + prefs.edit().putBoolean(KEY_ARCORE_ENABLED, isChecked).apply()); + + // Field of view slider + fovValue = findViewById(R.id.settings_fov_value); + SeekBar fovSeek = findViewById(R.id.settings_fov_seekbar); + fovSeek.setMax(FOV_MAX_PERCENT - FOV_MIN_PERCENT); + int percent = Math.round(prefs.getFloat(KEY_FOV_SCALE, DEFAULT_FOV_SCALE) * 100f); + percent = Math.max(FOV_MIN_PERCENT, Math.min(FOV_MAX_PERCENT, percent)); + fovSeek.setProgress(percent - FOV_MIN_PERCENT); + showFovPercent(percent); + fovSeek.setOnSeekBarChangeListener( + new SeekBar.OnSeekBarChangeListener() { + @Override + public void onProgressChanged(SeekBar bar, int progress, boolean fromUser) { + int p = FOV_MIN_PERCENT + progress; + showFovPercent(p); + if (fromUser) { + prefs.edit().putFloat(KEY_FOV_SCALE, p / 100f).apply(); + } + } + + @Override + public void onStartTrackingTouch(SeekBar bar) {} + + @Override + public void onStopTrackingTouch(SeekBar bar) {} + }); + + // Floor (height) mode + Spinner floorSpinner = findViewById(R.id.settings_floor_spinner); + ArrayAdapter adapter = + ArrayAdapter.createFromResource( + this, R.array.floor_mode_choices, android.R.layout.simple_spinner_item); + adapter.setDropDownViewResource(android.R.layout.simple_spinner_dropdown_item); + floorSpinner.setAdapter(adapter); + int mode = prefs.getInt(KEY_FLOOR_MODE, DEFAULT_FLOOR_MODE); + floorSpinner.setSelection(Math.max(0, Math.min(adapter.getCount() - 1, mode))); + floorSpinner.setOnItemSelectedListener( + new AdapterView.OnItemSelectedListener() { + @Override + public void onItemSelected( + AdapterView parent, View view, int position, long id) { + prefs.edit().putInt(KEY_FLOOR_MODE, position).apply(); + } + + @Override + public void onNothingSelected(AdapterView parent) {} + }); + } + + private void showFovPercent(int percent) { + fovValue.setText(percent + "%"); + } +} diff --git a/code/mobile/android/PhoneVR/app/src/main/res/layout/activity_init.xml b/code/mobile/android/PhoneVR/app/src/main/res/layout/activity_init.xml index a3da1607..65ca0509 100644 --- a/code/mobile/android/PhoneVR/app/src/main/res/layout/activity_init.xml +++ b/code/mobile/android/PhoneVR/app/src/main/res/layout/activity_init.xml @@ -7,6 +7,17 @@ tools:context=".InitActivity"> +