diff --git a/README.md b/README.md index b192f8c5..41cce56f 100644 --- a/README.md +++ b/README.md @@ -22,138 +22,82 @@ Use Steam VR-enabled applications with your phone as VR HMD (*Head-mounted displ | Issues | [`Github Issue Tracker`](https://github.com/PhoneVR-Developers/PhoneVR/issues) | | Discuss/Chat/Help/Real-Time Updates/Any - Discord | [`Discord Server Invite Link`](https://discord.com/invite/pNYtBNk) | +## ALVR v21 port with ARCore position tracking (this branch) + +This branch ports the Android client from ALVR v20 to ALVR v21, because newer SteamVR releases +are breaking v20 servers, and adds 6DoF position tracking with ARCore. + +* **ALVR v21:** the client talks to an ALVR **v21** streamer. + This client was built against `21.0.0-dev14` (ALVR commit `9f118394`). **v20 servers are not + compatible with this branch.** +* **Video:** Drame stutter present in the v20 client is + fixed. +* **Position tracking (6DoF):** on phones with ARCore, head *position* is tracked. Cardboard's rotation drift (yaw) is corrected against it with a complementary filter. Phones + without ARCore (or without Google Play Services for AR) run as 3DoF with the Cardboard tracker. + See [`ARCORE_NOTES.md`](./ARCORE_NOTES.md). +* **Status:** works, but not released yet. There are no APKs for this branch; you build it from source + (see [Building the v21 client](#building-the-v21-client)). The legacy PhoneVR server and the + `gvr` build flavour are not part of this branch. +* **Technical details and the problems found on the way:** [`MIGRATION_NOTES.md`](./MIGRATION_NOTES.md). + +Tested phones: + +| Phone | Android | GPU | ARCore | Result | +| --- | --- | --- | --- | --- | +| Oppo Reno 9 A | 14 | Adreno 619 | yes | Video, 6DoF position and drift-free yaw | +| AQUOS sense4 basic | 12 | Adreno 618 | yes | Works (ARCore is started only after video is flowing) | +| UMIDIGI A3S | 10 | PowerVR Rogue GE8300 | no | Works as 3DoF | + +Known limits: the yaw correction uses one fixed phone-to-ARCore alignment measured on a single phone, +so other phones may need a calibration that is not configurable yet; the floor height setting has not +been verified in a headset; and ARCore tracking is started again each time a stream starts, which +may cause a small position jump at that moment. + ## Contents +* [ALVR v21 port with ARCore (this branch)](#alvr-v21-port-with-arcore-position-tracking-this-branch) * [Requirements](#requirements) * [Installation](#installation) * [Advanced Configuration](#advanced-configuration) * [Development](#development) +* [Building the v21 client](#building-the-v21-client) * [Troubleshooting](#troubleshooting) * [Issue / Bug Reporting](#issue--bug-reporting) ## Requirements -A PC with *Windows 7 or above*, A smartphone with *Android 5.0 (Lollipop) or above* (_ALVR requires Android 8.0 or above_) with *OpenGL-ES 3.0 or above*, Steam and some SteamVR applications installed. +**For the ALVR v21 branch:** an Android 8.0+ phone with OpenGL ES 3.0+ and a hardware video decoder, and an ALVR v21 streamer on the PC. Position tracking additionally needs a phone [supported by ARCore](https://developers.google.com/ar/devices) with *Google Play Services for AR* installed; without it the app still runs, as 3DoF. ## Installation -PhoneVR(PVR) can be used with two servers ALVR and PVRServer. ALVR Server is recommended over PVRServer, both use different APIs. - -| Differences | ALVR Server | PhoneVR Server | -| ------------------------------------------------------------------------------ | -------------------------------------------- | -------------------------- | -| Server/Desktop Side | OpenVR API | OpenVR API | -| Mobile Side | Google Cardboard API | Google VR API (deprecated) | -| GUI | ✅ | ❌ | -| User friendly Installer | ✅ | ❌ | -| Serverside hardware(GPU) Encoding | ✅ | ❌ | -| Serverside software Encoding | ✅ | ✅ | -| Automatic Firewall setting | ✅ | ❌ | -| Linux Support (\*Partial\*) | ✅ | ❌ | -| Developers | [Alvr-org](https://github.com/alvr-org/ALVR) | PhoneVR | -| Supports other HMDs ? Quest etc. | ✅ | ❌ | -| Open-Source | ✅ | ✅ | -| Network throttling | ✅ | ❌ | -| Average Latency | 150ms | 1500ms | -| VRApplication Graphics manipulation
(Brightness, resolution, contrast etc.) | ✅ | (Partial) | -| Code backend | Rust + C++ | C++ | +The original PhoneVR(PVR) used with two servers ALVR and PVRServer. However, this fork uses ALVR Server only. ### A. Installation using ALVR Server -* Currently only **[ALVR v20.8.0](https://github.com/alvr-org/ALVR/releases/tag/v20.8.0)** (latest stable) is supported by PhoneVR. - * Older version - - PhoneVR v1.4.0-beta -> ALVR v20.7.1 - - PhoneVR v1.3.0-beta -> ALVR v20.6.1 - - PhoneVR v1.2.0-beta -> ALVR v20.5.0 - - PhoneVR v1.1.0-beta -> ALVR v20.2.1 - - PhoneVR v1.0.0-beta -> ALVR v20.1.0 +> **This branch (v21):** use an ALVR **v21** streamer, i.e. a build from ALVR's `master` branch (tested with `21.0.0-dev14`); + * Use the install guide of ALVR Server from [here](https://github.com/alvr-org/ALVR#install). ### B. Installation using Legacy PhoneVR Server -* Make sure you have Steam and SteamVR installed (To find SteamVR on steam, `Library -> Tools -> SteamVR`). -* Download latest release [`PhoneVR-Server-vX.X.X-xxxxx.zip`](https://github.com/PhoneVR-Developers/PhoneVR/releases/latest) of this repository. -* Copy the whole folder `PVRServer` in `driver` folder of zip file into your `SteamVR/drivers` folder. (Default Path: `C:\Program Files (x86)\Steam\steamapps\common\SteamVR\drivers`. -* (Only for PhoneVR Server) To **play**, **first open the Phone App**(SteamVR should also be closed), then run the game of your choice on PC. (Obviously, both PC and Mobile should be on same Network, preferably Wifi 5.0) +> Not available on the ALVR v21 branch. ### Common Instructions for both Servers * Install the Android Apk on your mobile from [`Releases`](https://github.com/PhoneVR-Developers/PhoneVR/releases/latest) page. - * There are two Apks on the release page, one with GoogleVR(GVR) SDK and other without GVR SDK. If you plan to use ALVR-Server only you can install `PhoneVR-vX.X.X.noGvr.apk`. + * The ALVR v21 branch has no release APKs yet; build it as described in [Building the v21 client](#building-the-v21-client). * Make sure that "Run in Background", "Auto Start"(Restart on Crash) permissions, if exists on your device(especially Xiaomi users), are given. Also make sure that any kind of 3rd party battery saver app dosen't kill PhoneVR when in background. -* After opening the installed app, Choose ALVR(recommended) or PhoneVR Server, according to the server you installed. ## Advanced Configuration ### for ALVR Server * This can be found [here](https://github.com/alvr-org/ALVR/wiki/Settings-guide) and [here](https://github.com/alvr-org/ALVR/wiki/Settings-tutorial). -### for Phone VR Server -* **Windows Driver Settings** - - Windows SteamVR Driver, auto loads with SteamVR, and gets `FrameTextures` from VRApplication(like SteamVR Home, VRChat, etc.) via SteamVR. These Textures are then Encoded and Streamed, at specified `game_fps`, to Mobile device using x264 Encoder. Some configurations of this encoder can be adjusted via, `pvrsettings.json` in default installation location `C:\Program Files (x86)\Steam\steamapps\common\SteamVR\drivers\PVRServer\`.
- **Default Settings :** - ```json - { - "enable" : true, - "game_fps" : 65, - "video_stream_port" : 15243, - "pose_stream_port" : 51423, - "pairing_port" : 33333, - "encoder" : { - "preset" : "ultrafast", - "tune" : "zerolatency", - "qp" : 20, - "profile" : "baseline" - } - } - ``` - Most settings are self-explanatory. Any change in settings will require SteamVR to be restarted for application. - * *"encoder"* : Encoder x264 Settings - - *"preset"* : Can be set to "ultrafast", "superfast", "veryfast", "faster", "fast", "medium", "slow", "slower", "veryslow" or "placebo" - Warning: the speed of these presets scales dramatically. Ultrafast is a full 100 times faster than placebo! - These presets affect the encoding speed. Using a slower preset gives you better compression, or quality per filesize, whereas faster presets give you worse compression. In general, you should just use the preset you can afford to wait for. - - - *"tune"* : Can be set to "film", "animation", "grain", "stillimage", "psnr", "ssim", "fastdecode", "zerolatency" - Tune affects the video quality(and size), film being the best quality and gradually decreasing to the right - - - *"profile"* : Can be set to "baseline", "main", "high", "high10", "high422", "high444" - Applies the restrictions of the given profile. Currently available profiles are specified above, from most to least restrictive. Does NOT guarantee that the given profile will be used: if the restrictions of "High" are applied to settings that are already Baseline-compatible, the stream will remain baseline. In short, it does not increase settings, only decrease them. - - - *"qp"* : Quantization packet constant, Range 0-51, 0-Lossless - - - *"rc_method"* : Rate Control Method, 1/2/3 - Constant QP (CQP) - 0; - Constant Rate factor (CRF) - 1; - Average Bitrate (ABR) - 2; - Default is 1 i.e., CRF with rf 24 (and rf_max 26) - - Other standard x264 settings, - - *"qcomp"* : Default : 0 - - *"keyint_max"* : Default : -1 - - *"intra_refresh"* : Default : False - - *"bitrate"* : Default : -1 (min 1000) - - Do only adjust when required, and keep a eye on `pvrlog.txt` file in default installation location `C:\Program Files (x86)\Steam\steamapps\common\SteamVR\drivers\PVRServer\logs\` for "*Skipped frame! Please re-tune the encoder parameters*" messages. If you are getting these messages in excess, you may wish to downgrade settings since windows does not seems to have enough processing power and resources to render with existing settings. - ## Development This Project is presently under testing. But, pull requests are welcome. -* Windows PhoneVR Driver: `/code/windows/PhoneVR.sln` - * Compiled/Tested on `Microsoft Visual Studio 2017` - * After building copy the `driver_PhoveVR.dll` and other files from `/build/[win32 or x64]/` to local driver folder. - * For Runtime debugging, You need to Attach `MSVS JIT debugger` to `vrserver.exe` (which actually loads the driver_PhonveVR.dll) - * For testing, this project has 2 Build Configs, Debug and Release. Debug has lots of debugging callouts to both local-driver-folder/pvrlog.txt and MSVS Debugger Output. - * Android App: App folder: `/code/mobile/android/PhoneVR` * Compiled/Tested on `Android Studio 4.0.1` * For testing, this project has 2 Build Configs, Debug and Release. Debug has lots of debugging callouts to logcat from both JAVA and JNI. + * (The ALVR v21 branch builds a single ALVR-based variant; the gvr/noGvr split described next applies to the v20 line.) * There are also two build flavours: gvr and noGvr. The gvr flavour is a superset of the noGvr flavour that adds support for legacy PhoneVR server with Google VR SDK (deprecated). * The source code of the gvr flavour is in app/src/gvr/ and app/src/main/. The code of the noGvr flavour is in app/src/noGvr/ and app/src/main/. Respectively, any shared code in app/src/main/ has to be tested with both flavours. * The prepare-alvr-deps.sh script, which needs to be run before building supports the optional "nogvr" argument. If you run the script without arguments, you can build both flavours. If you run the script with the "nogvr" argument, you can only build the noGvr flavour. - * PhoneVR is Android AVD compatible. - * PhoneVR Legacy Server networking - * PhoneVR with GVR APK when run on AVD automatically assigns 10.0.2.2 (default emu addr) to pcIP - * After starting the AVD Emulator, need to run `adb forward tcp:33333 tcp:33333`, in terminal/cmd, this will enable PVRServer -> Android Emulator based PVRAndroidApp communication. - * ALVR Server networking - * Similar to ALVR-Over-USB (https://github.com/alvr-org/ALVR/wiki/ALVR-wired-setup-(ALVR-over-USB)) - * Once `ALVR Streamer (PC) Configuration` is done from the above link. Then you should, in terminal/cmd, execute - * `adb forward tcp:9943 tcp:9943` - * `adb forward tcp:9944 tcp:9944` * External Vendor Libraries used (all Headers included in respective Projects): * Json v3.8.0 (https://github.com/nlohmann/json) (code\windows\libs\json) @@ -163,54 +107,42 @@ This Project is presently under testing. But, pull requests are welcome. * GoogleVR 1.200 (https://github.com/googlevr/gvr-android-sdk) (code\mobile\android\libraries\jni) * ACRA v5.7 (https://github.com/ACRA/acra) * ALVR (https://github.com/alvr-org/ALVR) (code\mobile\android\PhoneVR\ALVR) + * ARCore SDK 1.46.0 (https://github.com/google-ar/arcore-android-sdk) (ALVR v21 branch; C headers downloaded by `prepare-alvr-deps.sh`) + * wgpu-hal 25.0.2, patched (https://github.com/gfx-rs/wgpu) (ALVR v21 branch; patch in `code/mobile/android/PhoneVR/patches/`) * Code styling * Repo enforces clang-format v16.0.0 (.clang-format) for C/C++ and spotless (.editorconfig, google-java-format, ktfmt) for Java/Kotlin files * To autofix clang-format'ing issues devs can use `make format` * To autofix spotless format issues devs can goto PhoneVR android project folder and use `./gradlew spotlessApply` or use [spotless-gradle](https://plugins.jetbrains.com/plugin/18321-spotless-gradle) plugin for AndroidStudio/IDEs -## Troubleshooting -* Android App doesn’t connect to Windows Steam VR even after opening Phone App first and then SteamVR on windows - 1. Make sure that both the Windows and Android devices are in the same Local Network. (connected to the same router/wifi device) - 2. Sometimes, the port which PhoneVR use to connect win/mobile devices, also known as `Pairing Port (default :33333)`, might be used by other services on your devices(Windows/Android). Try changing the "Pairing Port" on `Android PhoneVR App` settings and `pairing_port` in `C:\Program Files\PhoneVR\pvrsettings.json` for `Windows PVR driver` and restart SteamVR. **Both Windows Pairing port and Android Pairing port should be the same**. Safe recommended port range : `30000 - 65535` - -* Android App automatically comes back to "Discovery"(Home/AppStart) page after some VR Application usage - - Check if `Android System Battery saver` or similar applications are killing the app when in background. Usually can be found is `Android Setting` -> `Application Manager` or `Application Settings` according to your Android device flavour/OEM. +## Building the v21 client + +Tested on Ubuntu (a VirtualBox VM) with JDK 17, Rust 1.97.1 and Android NDK 25.2.9519653. These steps +are condensed from [`MIGRATION_NOTES.md`](./MIGRATION_NOTES.md), which has the details. + +1. Install the Android SDK (a platform and build-tools must be installed) and NDK, and set + `JAVA_HOME`, `ANDROID_HOME` and `ANDROID_NDK_HOME`. +2. In `code/mobile/android/PhoneVR`, run `prepare-alvr-deps.sh`. It prepares the ALVR dependencies + (including `cargo xtask build-client-lib`) and downloads the ARCore C headers. +3. Patch `wgpu-hal` 25.0.2. Without this patch the lobby and video do not render, and some phones + (for example one with a PowerVR GE8300 GPU) crash. Download the crate from + `https://static.crates.io/crates/wgpu-hal/wgpu-hal-25.0.2.crate`, unpack it, and from inside the + unpacked folder run + `patch -p1 < /code/mobile/android/PhoneVR/patches/wgpu-hal-25.0.2-share-egl-context.patch`. + It makes the Rust GL context share the app's EGL context and turns off GL debug output. +4. In ALVR's top-level `Cargo.toml`, point `[patch.crates-io]` at the patched folder: + `wgpu-hal = { path = "" }`. + (The path used during development was absolute and machine-specific; use your own.) +5. Build the app with Gradle (Android Studio or `./gradlew assembleDebug`). -* Lag while using VR App on mobile - - Make sure you are not using debug variant of APK(unless you really want to). - - All relevant component's FPS are displayed on Mobile device when `Settings -> Debug` is checked. You can find out what the bottleneck component in the whole chain from VRApplication on windows to GoogleVR SDK renderer on Android is and resolve it. - -* Incase above mentioned things have not been helpful to you, then you might want to **Install Debug variant of APK** from `[PhoneVR.zip]/android/arm7/debug` which can help you/us to get more detailed logs. - -* **Diagnostics Stats Nomenclature (ONLY PhoneVR Server)** (in order from VRAppFrameCreation to MobileFrameDisplay):
- These stats can be enabled by checking `Settings`->`Stats` option. All the values must be stable and more or less equal to `game_fps`± 10 value in `pvrsettings.json`. -
Nomenclature - - ```css - --- @M - Mobile; @C - CPU/Desktop --- - ``` - ```css - @FPS(s) : - SR - StreamReceiver @M - D - Media Decoder @M - R - Frame Renderer @M - - cR - Frame Renderer @C - SS - Stream Sender @C - SW - Stream Writer @C - E - Media Encoder @C - VRa - VR Application FPS @C - - @Latency(s) : - tSS - Time Delay between RendererGotFrame and RendererRendered @C - tE - Time Delay between EncoderGotFrame and EncoderEncoded @C - tND - NetworkDelay - Time Delay between DataPacketSent from @C to DataPacketReceived @M - tSR - StreamReceiver - Time Delay between DataPacketSent from @C to DataSentToMediaDecoder @M - ``` +## Troubleshooting +* **(ALVR v21 branch)** The lobby freezes as soon as the stream starts, while SteamVR still shows head movement + - This happened on phones where the ARCore camera and the video decoder conflict. This branch starts ARCore only after video is flowing (`ARCORE_START_AFTER_VIDEO` in `alvr_main.cpp`). If you still see it, try turning ARCore off (last argument of `initializeNative` in `ALVRActivity.java`) and send the log. +* **(ALVR v21 branch)** The app crashes right after pressing the start button + - Capture `adb logcat -d | grep -a -E "FATAL|UnsatisfiedLinkError|Fatal signal|panicked|dlopen"` and check that the `wgpu-hal` patch (see [Building the v21 client](#building-the-v21-client)) was applied. +* **(ALVR v21 branch)** No position tracking + - Check the log for `ARCore: tracking`. If you see `could not create session`, the phone does not support ARCore or Google Play Services for AR is missing; the app then continues as 3DoF. - ![PhonveVR FPS](./.github/fps.jpg) -
## Issue / Bug Reporting @@ -238,9 +170,6 @@ For quick resolution you may want to add the following data along with your issu 3. Attach the `bugreport-xxx.zip` file in Discord or GitHub or mail to *"phonevr.crash@gmail.com"* **Common files required for all kinds issues,** -- If using PhoneVR Legacy server - * `pvrLog.txt` and/or `pvrDebugLog.txt` file(s) in default **windows** location `C:\Program Files (x86)\Steam\steamapps\common\SteamVR\drivers\PVRServer\logs` - * `pvrLog.txt` and/or `pvrDebugLog.txt` file(s) in default **android** location `.../Android/data/virtualis.phonevr/files/PVR/`. *Optionally* you can also attach `Log` from your `Settings page` on the app. - If using ALVR * ALVR logs from its server, are available in the GUI (you may which to enable Setting -> Logging -> [Log to disk](https://github.com/alvr-org/ALVR/wiki/Settings-guide#log-to-disk) if you want to keep saving the logs to file, even after alvr is closed * Screenshots / Videos are much more helpful for developers to understand the issues faster. diff --git a/code/mobile/android/PhoneVR/.gitignore b/code/mobile/android/PhoneVR/.gitignore index fa0251e2..525fd2db 100644 --- a/code/mobile/android/PhoneVR/.gitignore +++ b/code/mobile/android/PhoneVR/.gitignore @@ -48,3 +48,9 @@ google-services.json # Android Profiling *.hprof +arcore-android-sdk-main/ +/wgpu-hal-25.0.2/ +*.bak +*.orig +*.rej +signing.properties diff --git a/code/mobile/android/PhoneVR/ALVR b/code/mobile/android/PhoneVR/ALVR index 1d121fa7..9f118394 160000 --- a/code/mobile/android/PhoneVR/ALVR +++ b/code/mobile/android/PhoneVR/ALVR @@ -1 +1 @@ -Subproject commit 1d121fa7e0761ef26473c6b3f3fbd9b6051ce033 +Subproject commit 9f11839431f95f0df6764790d9a16f8e308a0d79 diff --git a/code/mobile/android/PhoneVR/app/CMakeLists.txt b/code/mobile/android/PhoneVR/app/CMakeLists.txt index 358126a1..f0db9551 100644 --- a/code/mobile/android/PhoneVR/app/CMakeLists.txt +++ b/code/mobile/android/PhoneVR/app/CMakeLists.txt @@ -1,7 +1,5 @@ cmake_minimum_required(VERSION 3.4.3) -#use rtti, exceptions for asio library ## -std=c++17 -#set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -g -frtti -fexceptions -Wc++11-narrowing -Wreorder") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++17 -g") set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS} -O3 -std=c++17 -g") set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS} -O3 -std=c++17 -g") @@ -9,36 +7,29 @@ set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS} -O3 -std=c++17 -g") set(ANDROID_STL c++_static) set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY BOTH) -file(GLOB_RECURSE LIB_SRC - ../../../../common/libs/ifaddrs/*.c - ../../../../common/libs/json/single_include/*.hpp -) - add_library(native-lib-alvr SHARED src/main/cpp/alvr_main.cpp - ${LIB_SRC} +) + +# Import the ARCore (Google Play Services for AR) library. +add_library(arcore SHARED IMPORTED) +set_target_properties(arcore PROPERTIES IMPORTED_LOCATION + ${ARCORE_LIBPATH}/${ANDROID_ABI}/libarcore_sdk_c.so + INTERFACE_INCLUDE_DIRECTORIES ${ARCORE_INCLUDE} ) set(libs_dir ${CMAKE_CURRENT_SOURCE_DIR}/libraries) set(alvr_build_dir ${CMAKE_CURRENT_SOURCE_DIR}/../ALVR/build/alvr_client_core) target_include_directories(native-lib-alvr - PUBLIC ../../../../common/libs/asio/asio/include - PUBLIC ../../../../common/libs/eigen/Eigen - PUBLIC ../../../../common/libs/ifaddrs - PUBLIC ../../../../common/libs/json/single_include/ - # ALVR Headers PUBLIC ${alvr_build_dir} PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/../cardboard + PUBLIC ${ARCORE_INCLUDE} ) target_link_libraries(native-lib-alvr - log android EGL GLESv3 mediandk + log android EGL GLESv3 mediandk arcore ${alvr_build_dir}/${ANDROID_ABI}/libalvr_client_core.so ${libs_dir}/jni/${ANDROID_ABI}/libGfxPluginCardboard.so ) - -IF(${HAVEGVR}) -add_subdirectory(src/gvr) -ENDIF() diff --git a/code/mobile/android/PhoneVR/app/build.gradle b/code/mobile/android/PhoneVR/app/build.gradle index 399bcf30..6f6d041b 100644 --- a/code/mobile/android/PhoneVR/app/build.gradle +++ b/code/mobile/android/PhoneVR/app/build.gradle @@ -7,16 +7,35 @@ apply plugin: 'com.google.gms.google-services' apply from: '../versioning.gradle' apply plugin: "com.diffplug.spotless" +def arcore_libpath = "${rootProject.layout.buildDirectory.get()}/arcore-native" + +configurations { natives } def keystorePropertiesFile = rootProject.file("keystore.properties") def keystoreProperties = new Properties() keystoreProperties.load(new FileInputStream(keystorePropertiesFile)) def defaultALVRFlavour = "--release" +// Part of the APK file name: PhoneVR-v--.apk +// e.g. PhoneVR-v2.0.0-beta-ARcore-v21-release.apk (VERSION_NAME is set in ../version.properties) +def releaseLabel = "ARcore-v21" + android { signingConfigs { releaseconfig { - if( System.getenv('GITHUB_EVENT_NAME') == 'pull_request' || System.getenv("Store") == null || System.getenv("Key") == null ) { + // Your own release key. signing.properties is NOT committed (it is in .gitignore) and + // holds: storeFile, storePassword, keyAlias, keyPassword. + def localSigningFile = rootProject.file("signing.properties") + if (localSigningFile.exists()) { + def localSigning = new Properties() + localSigningFile.withInputStream { localSigning.load(it) } + storeFile file(localSigning['storeFile']) + storePassword localSigning['storePassword'] + keyAlias localSigning['keyAlias'] + keyPassword localSigning['keyPassword'] + println "[PVR] Configuring releaseconfig with your own key from signing.properties" + } + else if( System.getenv('GITHUB_EVENT_NAME') == 'pull_request' || System.getenv("Store") == null || System.getenv("Key") == null ) { // If its a PR Supply a dummy KeySigningConfig // Or in Local env if Store and Keys are not present in ENV Vars do the same storeFile file("PhoneVRKeyStorePR.jks") @@ -24,6 +43,7 @@ android { keyAlias "PhoneVRKeyPR" keyPassword "test1234" println "[PVR] Configuring releaseconfigs for testing a PR..." + println "[PVR] WARNING: this APK is signed with the public TEST key (no signing.properties). Do not share it." } else { storeFile file(keystoreProperties['storeFile']) @@ -41,6 +61,11 @@ android { applicationId "viritualisres.phonevr" minSdkVersion 24 targetSdkVersion 33 + + ndk { + abiFilters "arm64-v8a" + } + versionCode buildVersionCode() versionName buildVersionName() @@ -50,6 +75,8 @@ android { externalNativeBuild { cmake { cppFlags "-std=c++17 -fexceptions -frtti" //-DANDROID_USE_LEGACY_TOOLCHAIN_FILE=ON" + arguments "-DARCORE_LIBPATH=${arcore_libpath}/jni", + "-DARCORE_INCLUDE=${project.rootDir}/arcore-android-sdk-main/libraries/include" } } @@ -77,6 +104,12 @@ android { defaultALVRFlavour = "" } } + // APK file name, e.g. PhoneVR-v2.0.0-beta-ARcore-v21-release.apk + applicationVariants.all { variant -> + variant.outputs.all { + outputFileName = "PhoneVR-v${defaultConfig.versionName}-${releaseLabel}-${variant.buildType.name}.apk" + } + } buildFeatures { buildConfig = true // TODO: Remove BuildConfig, Gradle suggests this usage is inefficient > v8 disabled by default viewBinding true @@ -93,7 +126,8 @@ android { } noGvr { dimension "gvr" - versionNameSuffix "-nogvr" + //versionNameSuffix "-nogvr" + versionNameSuffix "" } } sourceSets { @@ -167,13 +201,16 @@ dependencies { gvrImplementation 'com.google.vr:sdk-base:1.200.0@aar' implementation 'com.google.cardboard.sdk:cardboard-sdk:1.17.0@aar' - implementation 'io.github.zxing-cpp:android:2.3.0-SNAPSHOT' + implementation 'io.github.zxing-cpp:android:2.3.0' implementation 'com.google.protobuf:protobuf-javalite:3.19.4' implementation 'androidx.camera:camera-core:1.3.4' implementation 'androidx.camera:camera-view:1.3.4' implementation 'androidx.camera:camera-lifecycle:1.3.4' implementation 'androidx.camera:camera-camera2:1.3.4' + implementation 'com.google.ar:core:1.46.0' + natives 'com.google.ar:core:1.46.0' + def acraVersion = '5.7.0' implementation "ch.acra:acra-mail:$acraVersion" implementation "ch.acra:acra-dialog:$acraVersion" @@ -199,6 +236,13 @@ task extractNdk(type: Copy) { include "jni/**/libgvr.so" } } + configurations.natives.files.each { f -> + copy { + from zipTree(f) + into arcore_libpath + include "jni/**/*" + } + } } task deleteNdk(type: Delete) { diff --git a/code/mobile/android/PhoneVR/app/src/main/AndroidManifest.xml b/code/mobile/android/PhoneVR/app/src/main/AndroidManifest.xml index c193327f..528542fc 100644 --- a/code/mobile/android/PhoneVR/app/src/main/AndroidManifest.xml +++ b/code/mobile/android/PhoneVR/app/src/main/AndroidManifest.xml @@ -12,6 +12,7 @@ + + + + + +#include #include #include +#include +#include #include #include #include +#include #include #include #include -#include "nlohmann/json.hpp" +#include "head_sensor_source.h" +#include "neck_model.h" #include "utils.h" -using namespace nlohmann; + +#ifndef LOG +#define LOG(...) __android_log_print(ANDROID_LOG_INFO, "ALVR_EGLCHK", __VA_ARGS__) +#endif +#define EGLCHK(where) LOG("EGLCHK " where " ctx=%p surf=%p", eglGetCurrentContext(), eglGetCurrentSurface(EGL_DRAW)) + +struct EglStateGuard { + EGLDisplay dpy = eglGetCurrentDisplay(); + EGLContext ctx = eglGetCurrentContext(); + EGLSurface draw = eglGetCurrentSurface(EGL_DRAW); + EGLSurface read = eglGetCurrentSurface(EGL_READ); + ~EglStateGuard() { + if (ctx != EGL_NO_CONTEXT && eglGetCurrentContext() != ctx) + eglMakeCurrent(dpy, draw, read, ctx); + } +}; + +// TEMP DEBUG: burn texture names before creating the lobby swapchain, so the names handed to the +// ALVR library (17,18 instead of 1,2) can't coincide with names its own EGL context already uses. +// Remove once the context-sharing question is settled. +#define PVR_TEXTURE_NAME_TEST 1 + +// The ALVR graphics code aborts the process when it sees a GL error. glGetError() is sticky, so +// an error left behind by app/Cardboard code would be blamed on the library call. Drain (and +// log) any pending errors before each library call. +static void drainGlErrors(const char *where) { + GLenum e; + int n = 0; + while ((e = glGetError()) != GL_NO_ERROR && n++ < 16) + LOG("Pending GL error 0x%x before %s", e, where); +} + +// One-time GL driver info, to check feature support (external images) when debugging. +static void logGlInfo() { + const char *ver = (const char *) glGetString(GL_VERSION); + const char *ren = (const char *) glGetString(GL_RENDERER); + const char *ext = (const char *) glGetString(GL_EXTENSIONS); + LOG("GLINFO version=%s renderer=%s", ver ? ver : "?", ren ? ren : "?"); + LOG("GLINFO image_external_essl3=%d image_external=%d", + ext && strstr(ext, "GL_OES_EGL_image_external_essl3") != nullptr, + ext && strstr(ext, "GL_OES_EGL_image_external") != nullptr); +} + +// The header says later DecoderConfig events are ignored until reconnection. +static bool g_decoderCreated = false; + +// alvr_*_opengl() calls in this ALVR version can leave the thread with no current EGL context. +// Run them under a guard so the app's context/surface is rebound right after each call. +#define ALVR_GL_CALL(call) \ + do { \ + drainGlErrors(#call); \ + EglStateGuard _eglGuard; \ + call; \ + } while (0) + uint64_t HEAD_ID = alvr_path_string_to_id("/user/head"); +// v21 removed the alvr_get_head_prediction_offset_ns() C export (it's now +// ClientCoreContext::get_total_prediction_offset(), a Rust-only method used internally by +// client_openxr). There is currently no C-ABI replacement, so this is a fixed heuristic in the +// meantime. +// TODO(upstream): propose re-exposing get_total_prediction_offset() via c_api.rs so native +// clients don't have to guess this. +const uint64_t ESTIMATED_PREDICTION_OFFSET_NS = 30e6; + // Note: the Cardboard SDK cannot estimate display time and an heuristic is used instead. -const uint64_t VSYNC_QUEUE_INTERVAL_NS = 50e6; +// Tested on the Oppo Reno 9 A: 50 ms made the streamed picture overshoot when a fast head turn +// stopped (felt over-sensitive); 30 ms removes that. Candidate for per-phone calibration. +const uint64_t VSYNC_QUEUE_INTERVAL_NS = 30e6; + +// Client-side upscaling (ALVR's Snapdragon Game Super Resolution pass). The server streams a +// smaller picture, the decoder works on that, and the client scales it up to (about) the phone's +// real per-eye size. Only worth turning on when the stream is smaller than the display, e.g. to +// save Wi-Fi bandwidth or decoder load: set the server's video resolution lower in the dashboard. +// The upscale factor is worked out from the stream size, so there is nothing else to tune. +constexpr bool ENABLE_UPSCALING = true; +constexpr float UPSCALING_MAX_FACTOR = 3.0f; // the same limit as ALVR's own slider +// Same defaults as ALVR's own settings. +constexpr bool UPSCALING_EDGE_DIRECTION = true; +constexpr float UPSCALING_EDGE_THRESHOLD = 4.0f; +constexpr float UPSCALING_EDGE_SHARPNESS = 2.0f; + +// Zoom of the streamed picture. The server renders with the field of view we report; reporting a +// narrower one than the Cardboard lens really shows gives a bigger, closer-looking picture with +// more pixels per degree (so text is easier to read). 1.0 = exactly what the lens geometry says. +// The value scales the tangent of the half-angles, so 0.8 = about 25% zoom. The lobby is not +// affected. Too small a value makes straight lines bend slightly toward the edges. +// Tested on the Oppo Reno 9 A against the native client, which looked about 25% more zoomed in. +// Default; the app's Settings screen can change it (see setStreamFovScaleNative). +constexpr float STREAM_FOV_SCALE = 0.8f; const float FLOOR_HEIGHT = 1.5; const int MAXIMUM_TRACKING_FRAMES = 360; +// ---- ARCore tuning (values carried over from the v20 branch; tuned on one phone only) ---- +// Use ARCore's position as the head position (otherwise position stays at the origin: 3DoF). +// Some phones (seen on an AQUOS sense4 basic, Adreno 618, Android 12) stop delivering decoded +// video frames while the ARCore camera session is running. With this set, ARCore is only +// started once the video decoder has produced a few frames, and it is paused again when the +// stream stops. Set to false to start ARCore together with the app, as before. +constexpr bool ARCORE_START_AFTER_VIDEO = true; +constexpr int ARCORE_START_AFTER_VIDEO_FRAMES = 10; + +constexpr bool ARCORE_USE_POSITION = true; +// Slowly pull Cardboard's drifting orientation toward ARCore's (complementary filter). +constexpr bool ARCORE_USE_ORIENTATION_CORRECTION = true; +// How fast the correction follows ARCore. Larger = slower and smoother. +constexpr float ARCORE_CORRECTION_TIME_CONSTANT_S = 3.0f; +// A sample is ignored as a bad measurement when it disagrees with the orientation we are +// currently showing (Cardboard plus the correction) by more than this. How much drift Cardboard +// has built up in total does not matter, only the disagreement left after the correction. +constexpr float ARCORE_MAX_RESIDUAL_DEG = 15.0f; +// If samples keep disagreeing for this long, ARCore has probably moved its world frame (tracking +// was lost and regained, or it relocalized), so the new value is accepted... +constexpr float ARCORE_REACQUIRE_AFTER_S = 1.5f; +// ...and the correction follows it with this faster time constant until the two agree again. +constexpr float ARCORE_REACQUIRE_TIME_CONSTANT_S = 1.0f; +// Ignore the update after a gap longer than this (pause, lifecycle change, debugger). +constexpr uint64_t ARCORE_MAX_CORRECTION_GAP_NS = 500000000ULL; +// Fixed rotation between ARCore's display-oriented camera frame and Cardboard's frame +// (landscape-left). Found by calibration on one phone; other phones may need other values. +const AlvrQuat ARCORE_TO_CARDBOARD = {-0.0057f, -0.0031f, 0.7076f, 0.7066f}; + +// Which axis is "up" in the orientation this file works with (0 = x, 1 = y, 2 = z). The yaw +// correction is a rotation about this axis. The log prints a "tilt disagreement" that should stay +// small (a few degrees); if it is large, this axis is probably wrong. +constexpr int ARCORE_WORLD_UP_AXIS = 1; + +// ---- Own head orientation (replaces Cardboard's gyro tracker) ---- +// Mahony filter plus a stillness-based gyro bias estimator (head_orientation_filter.h), fed by +// head_sensor_source.h. Cardboard's tracker drifts in yaw because it never measures the gyro's +// constant offset; this does. The ARCore yaw correction below works on top of it unchanged. +// Set to false to go back to Cardboard's tracker. Falls back to it until the first sensor data +// arrives (or if the phone has no gyroscope). +constexpr bool USE_OWN_HEAD_FILTER = true; +// Once a second, while streaming: compare the own filter with Cardboard's tracker in the log. +constexpr bool HEAD_FILTER_DEBUG_LOG = true; + +// ---- Neck model (seated 3DoF) ---- +// Without ARCore there is no position, so the view is glued to one point. With this on, pitching +// or rolling the head moves the eyes a few centimetres, as it does on a real neck (see +// neck_model.h). Only used when ARCore is off; ARCore's real position replaces it. Turning the +// head left or right is deliberately ignored. +constexpr bool NECK_MODEL_ENABLED = true; +// Where the eyes are relative to the neck pivot, in metres (rough adult average; per-user +// setting later). +constexpr float NECK_EYE_UP_M = 0.09f; +constexpr float NECK_EYE_FORWARD_M = 0.08f; + +// ---- ARCore head height ---- +// How the vertical position is derived: +// 0 = relative to where ARCore started (height starts at 0, as in the earlier builds). +// 1 = height above the detected floor (the v20 behaviour). Until a floor is found it is the +// same as 0, so the height jumps the moment the floor is first detected. +// 2 = like 1, but until a floor is found it assumes the headset started +// ARCORE_ASSUMED_START_HEIGHT_M above the floor, and it eases toward the measured floor +// height instead of jumping. +// Default; the app's Settings screen can change it (see setFloorModeNative). +constexpr int ARCORE_HEIGHT_MODE = 2; +// Mode 2 only: assumed height above the floor at the start, before any floor has been detected. +constexpr float ARCORE_ASSUMED_START_HEIGHT_M = 1.5f; +// Mode 2 only: how slowly the height eases toward the measured floor. Larger = slower. +constexpr float ARCORE_FLOOR_BLEND_TIME_CONSTANT_S = 2.0f; +// Added to the height in every mode. Use it to tune the height by hand (negative = lower). +constexpr float ARCORE_HEIGHT_OFFSET_M = 0.0f; +// A newly detected plane has to be this much lower than the current floor to replace it. +constexpr float ARCORE_FLOOR_REANCHOR_MARGIN_M = 0.03f; + struct NativeContext { JavaVM *javaVm = nullptr; jobject javaContext = nullptr; CardboardHeadTracker *headTracker = nullptr; + phonevr::HeadOrientationFilter headFilter; + phonevr::HeadSensorSource headSensors; + float neckLastYawRad = 0.f; // heading held while looking straight up/down; under poseMutex CardboardLensDistortion *lensDistortion = nullptr; CardboardDistortionRenderer *distortionRenderer = nullptr; + // ARCore (positional tracking). ARCore calls that touch the session must not run + // concurrently, so every use of arSession is guarded by arMutex. ArSession_update() must + // additionally run on the render thread, with the app's EGL context current. + bool arcoreEnabled = false; + // Settings chosen on the app's Settings screen (the constants above are the defaults). + float streamFovScale = STREAM_FOV_SCALE; + int heightMode = ARCORE_HEIGHT_MODE; + bool arSessionRunning = false; // ArSession_resume() done and not yet paused; under arMutex + int videoFramesSeen = 0; // decoded frames since the stream started (render thread) + ArSession *arSession = nullptr; + ArFrame *arFrame = nullptr; + // Anchor on the lowest horizontal plane found so far (taken to be the floor). An anchor keeps + // following the floor when ARCore adjusts its world frame. Render thread only (arMutex). + ArAnchor *arFloorAnchor = nullptr; + // Mode 2: how far above the floor ARCore's origin is, eased toward the measured value. + float floorOffsetM = ARCORE_ASSUMED_START_HEIGHT_M; + uint64_t lastFloorBlendNs = 0; + GLuint arTexture = 0; + std::mutex arMutex; + + // Pose state written by the render thread (updateARCoreTracking) and read by any thread + // that calls getPose(), for example the input thread. Guarded by poseMutex. + std::mutex poseMutex; + bool arPositionValid = false; + float arPosition[3] = {0.f, 0.f, 0.f}; + // Yaw (rotation about the world's up axis) added to Cardboard's orientation, in radians. + float yawCorrectionRad = 0.f; + uint64_t lastCorrectionNs = 0; + uint64_t disagreeSinceNs = 0; // when ARCore started disagreeing with the correction, or 0 + int screenWidth = 0; int screenHeight = 0; @@ -40,19 +238,30 @@ struct NativeContext { bool streaming = false; std::thread inputThread; - // Une one texture per eye, no need for swapchains. + // Use one texture per eye, no need for swapchains. GLuint lobbyTextures[2] = {0, 0}; GLuint streamTextures[2] = {0, 0}; float eyeOffsets[2] = {0.0, 0.0}; AlvrFov fovArr[2] = {}; - AlvrViewParams viewParams[2] = {}; AlvrDeviceMotion deviceMotion = {}; + // Cache of the most recently matched frame metadata, since alvr_report_compositor_start() + // can legitimately return false (no match for this exact frame), in which case the caller + // is expected to reuse the previous metadata. + bool hasFrameMetadata = false; + AlvrVideoFrameMetadata lastFrameMetadata = {}; + + // The last decoded video frame, kept so it can be shown again (re-oriented to the current + // head pose) on display refreshes where no new video frame has arrived. The buffer belongs + // to the decoder, so this is cleared whenever the decoder is created or destroyed or a + // stream starts or stops. + void *lastStreamBuffer = nullptr; + uint64_t lastStreamTimestampNs = 0; + NativeContext() { - memset(&fovArr, 0, (sizeof(fovArr)) / sizeof(int)); - memset(&viewParams, 0, (sizeof(viewParams)) / sizeof(int)); - memset(&deviceMotion, 0, (sizeof(deviceMotion)) / sizeof(int)); + memset(&fovArr, 0, sizeof(fovArr)); + memset(&deviceMotion, 0, sizeof(deviceMotion)); } }; @@ -105,38 +314,500 @@ AlvrFov getFov(CardboardEye eye) { return fov; } -AlvrPose getPose(uint64_t timestampNs) { - AlvrPose pose = {}; +// The field of view reported to the server, i.e. the one the stream is rendered with. The lobby +// keeps using getFov() directly. See STREAM_FOV_SCALE. +AlvrFov getStreamFov(CardboardEye eye) { + AlvrFov fov = getFov(eye); + fov.left = atanf(tanf(fov.left) * CTX.streamFovScale); + fov.right = atanf(tanf(fov.right) * CTX.streamFovScale); + fov.up = atanf(tanf(fov.up) * CTX.streamFovScale); + fov.down = atanf(tanf(fov.down) * CTX.streamFovScale); + return fov; +} + +// Quaternion helpers for the orientation correction (same maths as the v20 branch). +static AlvrQuat quatMultiply(const AlvrQuat &a, const AlvrQuat &b) { + return {a.w * b.x + a.x * b.w + a.y * b.z - a.z * b.y, + a.w * b.y - a.x * b.z + a.y * b.w + a.z * b.x, + a.w * b.z + a.x * b.y - a.y * b.x + a.z * b.w, + a.w * b.w - a.x * b.x - a.y * b.y - a.z * b.z}; +} + +// Inverse of a unit quaternion. +static AlvrQuat quatInverse(const AlvrQuat &q) { return {-q.x, -q.y, -q.z, q.w}; } +static AlvrQuat normalizeQuat(AlvrQuat q) { + float len = std::sqrt(q.x * q.x + q.y * q.y + q.z * q.z + q.w * q.w); + if (len < 1e-6f) { + return {0.f, 0.f, 0.f, 1.f}; + } + return {q.x / len, q.y / len, q.z / len, q.w / len}; +} + +// q and -q are the same rotation, so use abs(w). +static float quatAngleDegrees(const AlvrQuat &q) { + float w = std::clamp(std::abs(q.w), 0.0f, 1.0f); + return 2.0f * std::acos(w) * 180.0f / (float) M_PI; +} + +// Wraps an angle to the range [-pi, pi]. +static float wrapRad(float angle) { + const float twoPi = 2.0f * (float) M_PI; + while (angle > (float) M_PI) { + angle -= twoPi; + } + while (angle < -(float) M_PI) { + angle += twoPi; + } + return angle; +} + +// A rotation about the world's up axis. +static AlvrQuat worldYawQuat(float angleRad) { + float s = std::sin(angleRad * 0.5f); + float c = std::cos(angleRad * 0.5f); + float axis[3] = {0.f, 0.f, 0.f}; + axis[ARCORE_WORLD_UP_AXIS] = s; + return {axis[0], axis[1], axis[2], c}; +} + +// Cardboard's own tracker, in the same convention the rest of this file uses. +static AlvrQuat cardboardTrackerOrientationAt(uint64_t timestampNs) { float pos[3]; float q[4]; CardboardHeadTracker_getPose(CTX.headTracker, (int64_t) timestampNs, kLandscapeLeft, pos, q); + return normalizeQuat(inverseQuat(AlvrQuat{q[0], q[1], q[2], q[3]})); +} - auto inverseOrientation = AlvrQuat{q[0], q[1], q[2], q[3]}; - pose.orientation = inverseQuat(inverseOrientation); +// The gyro-based orientation at the given time (before any ARCore yaw correction): our own +// filter (USE_OWN_HEAD_FILTER), or Cardboard's tracker until the filter has data. +static AlvrQuat cardboardOrientationAt(uint64_t timestampNs) { + if (USE_OWN_HEAD_FILTER && CTX.headFilter.ready()) { + phonevr::Quatd o = CTX.headFilter.orientationAt((int64_t) timestampNs); + return normalizeQuat(AlvrQuat{(float) o.x, (float) o.y, (float) o.z, (float) o.w}); + } + return cardboardTrackerOrientationAt(timestampNs); +} + +// Yaw (radians, about +y) of the direction the head faces (-z), for the log check below. +static float facingYawRad(const AlvrQuat &q) { + float fx = -2.f * (q.x * q.z + q.w * q.y); + float fz = -(1.f - 2.f * (q.x * q.x + q.y * q.y)); + return std::atan2(-fx, -fz); +} + +// Angle (degrees) between the two orientations' "up" directions, as seen from the head. This is +// pitch/roll only, so it ignores yaw. Close to 0 when the own filter's axes match Cardboard's. +static float upVectorAngleDeg(const AlvrQuat &a, const AlvrQuat &b) { + auto up = [](const AlvrQuat &q, float v[3]) { + v[0] = 2.f * (q.x * q.y + q.w * q.z); + v[1] = 1.f - 2.f * (q.x * q.x + q.z * q.z); + v[2] = 2.f * (q.y * q.z - q.w * q.x); + }; + float va[3], vb[3]; + up(a, va); + up(b, vb); + float dot = std::clamp(va[0] * vb[0] + va[1] * vb[1] + va[2] * vb[2], -1.f, 1.f); + return std::acos(dot) * 180.f / (float) M_PI; +} +// Head pose, in world/tracking space (not head-relative). Safe to call from any thread. +// +// Orientation: the fast gyro orientation (own filter, or Cardboard's), turned about the vertical by a slowly changing +// yaw correction that the render thread computes from ARCore (updateARCoreTracking). Gyro drift +// is a turn about the vertical, so this is what removes it. +// Position: ARCore's last known position, or the origin until ARCore has tracked (3DoF). +AlvrPose getPose(uint64_t timestampNs) { + AlvrPose pose = {}; + + AlvrQuat orientation = cardboardOrientationAt(timestampNs); + + { + std::lock_guard lock(CTX.poseMutex); + + if (ARCORE_USE_ORIENTATION_CORRECTION) { + orientation = + normalizeQuat(quatMultiply(worldYawQuat(CTX.yawCorrectionRad), orientation)); + } + + if (ARCORE_USE_POSITION && CTX.arPositionValid) { + for (int i = 0; i < 3; i++) { + pose.position[i] = CTX.arPosition[i]; + } + } else if (NECK_MODEL_ENABLED && !CTX.arcoreEnabled) { + float offset[3]; + phonevr::neckModelOffset(orientation.x, orientation.y, orientation.z, orientation.w, + NECK_EYE_UP_M, NECK_EYE_FORWARD_M, CTX.neckLastYawRad, + offset); + for (int i = 0; i < 3; i++) { + pose.position[i] = offset[i]; + } + } + } + + pose.orientation = orientation; return pose; } -void updateViewConfigs(uint64_t targetTimestampNs = 0) { - if (!targetTimestampNs) - targetTimestampNs = GetBootTimeNano() + alvr_get_head_prediction_offset_ns(); +// Height (y) of an anchor in ARCore's world frame. +static float anchorHeight(const ArAnchor *anchor) { + ArPose *pose = nullptr; + ArPose_create(CTX.arSession, nullptr, &pose); + ArAnchor_getPose(CTX.arSession, anchor, pose); + float raw[7] = {}; + ArPose_getPoseRaw(CTX.arSession, pose, raw); + ArPose_destroy(pose); + return raw[5]; +} + +// Looks for the floor: the lowest horizontal, upward-facing plane ARCore has found so far. When +// one is known, returns true and sets *outFloorY to the floor's height in ARCore's world frame +// (read from an anchor, so it follows any adjustment ARCore makes to that frame). +// Render thread only, with arMutex held and a successful ArSession_update behind it. +static bool updateFloorAnchor(float *outFloorY) { + ArTrackableList *planes = nullptr; + ArTrackableList_create(CTX.arSession, &planes); + ArFrame_getUpdatedTrackables(CTX.arSession, CTX.arFrame, AR_TRACKABLE_PLANE, planes); + int32_t planeCount = 0; + ArTrackableList_getSize(CTX.arSession, planes, &planeCount); + + // Is the current floor anchor still usable? + float floorY = 0.0f; + bool haveFloor = false; + if (CTX.arFloorAnchor != nullptr) { + ArTrackingState anchorState; + ArAnchor_getTrackingState(CTX.arSession, CTX.arFloorAnchor, &anchorState); + if (anchorState == AR_TRACKING_STATE_STOPPED) { + // ARCore has given up on this anchor, so forget it. + ArAnchor_release(CTX.arFloorAnchor); + CTX.arFloorAnchor = nullptr; + } else if (anchorState == AR_TRACKING_STATE_TRACKING) { + floorY = anchorHeight(CTX.arFloorAnchor); + haveFloor = true; + } + } + + for (int32_t i = 0; i < planeCount; i++) { + ArTrackable *trackable = nullptr; + ArTrackableList_acquireItem(CTX.arSession, planes, i, &trackable); + if (trackable == nullptr) { + continue; + } + + ArTrackingState planeState; + ArTrackable_getTrackingState(CTX.arSession, trackable, &planeState); + ArPlaneType planeType; + ArPlane_getType(CTX.arSession, ArAsPlane(trackable), &planeType); + + if (planeState == AR_TRACKING_STATE_TRACKING && + planeType == AR_PLANE_HORIZONTAL_UPWARD_FACING) { + ArPose *planePose = nullptr; + ArPose_create(CTX.arSession, nullptr, &planePose); + ArPlane_getCenterPose(CTX.arSession, ArAsPlane(trackable), planePose); + float planeRaw[7] = {}; + ArPose_getPoseRaw(CTX.arSession, planePose, planeRaw); + float planeY = planeRaw[5]; + + // The floor is the lowest such plane (tables and shelves are higher). + if (!haveFloor || planeY < floorY - ARCORE_FLOOR_REANCHOR_MARGIN_M) { + // Anchor at the plane's position only, without its rotation. (Poses are created + // from {qx, qy, qz, qw, tx, ty, tz}.) + const float translationOnlyRaw[7] = { + 0.f, 0.f, 0.f, 1.f, planeRaw[4], planeRaw[5], planeRaw[6]}; + ArPose *translationOnly = nullptr; + ArPose_create(CTX.arSession, translationOnlyRaw, &translationOnly); + ArAnchor *newAnchor = nullptr; + if (ArTrackable_acquireNewAnchor( + CTX.arSession, trackable, translationOnly, &newAnchor) == AR_SUCCESS) { + if (CTX.arFloorAnchor != nullptr) { + ArAnchor_detach(CTX.arSession, CTX.arFloorAnchor); + ArAnchor_release(CTX.arFloorAnchor); + } + CTX.arFloorAnchor = newAnchor; + info("ARCore: floor %s at y=%.3f", + haveFloor ? "lowered" : "found", + planeY); + floorY = planeY; + haveFloor = true; + } + ArPose_destroy(translationOnly); + } + ArPose_destroy(planePose); + } + ArTrackable_release(trackable); + } + ArTrackableList_destroy(planes); + + if (haveFloor) { + *outFloorY = floorY; + } + return haveFloor; +} + +// Updates ARCore once per render frame and publishes what getPose() needs: the ARCore position +// and the orientation correction. Must be called from the render thread, with the app's EGL +// context current (ArSession_update requires it). +// Starts the ARCore camera session (no-op if it is already running or ARCore is off). +static void startARCoreSession(const char *why) { + std::lock_guard lock(CTX.arMutex); + if (!CTX.arcoreEnabled || CTX.arSession == nullptr || CTX.arSessionRunning) { + return; + } + ArStatus arStatus = ArSession_resume(CTX.arSession); + if (arStatus != AR_SUCCESS) { + info("ARCore ERROR: failed to resume session (status %d)", (int) arStatus); + return; + } + CTX.arSessionRunning = true; + info("ARCore: camera session started (%s)", why); +} + +// Pauses the ARCore session, which releases the camera (no-op if it is not running). +static void stopARCoreSession(const char *why) { + { + std::lock_guard lock(CTX.arMutex); + if (!CTX.arcoreEnabled || CTX.arSession == nullptr || !CTX.arSessionRunning) { + return; + } + ArSession_pause(CTX.arSession); + CTX.arSessionRunning = false; + } + { + std::lock_guard lock(CTX.poseMutex); + CTX.arPositionValid = false; + } + info("ARCore: camera session paused (%s)", why); +} + +static void updateARCoreTracking() { + if (!CTX.arcoreEnabled || CTX.arTexture == 0) { + return; + } + if (eglGetCurrentContext() == EGL_NO_CONTEXT) { + info("ARCore ERROR: updateARCoreTracking called without an EGL context"); + return; + } + + std::lock_guard lock(CTX.arMutex); + if (CTX.arSession == nullptr || CTX.arFrame == nullptr || !CTX.arSessionRunning) { + return; + } + + static uint64_t lastLogNs = 0; + uint64_t nowNs = GetBootTimeNano(); + bool shouldLog = nowNs - lastLogNs >= 1000000000ULL; + + ArStatus status = ArSession_update(CTX.arSession, CTX.arFrame); + if (status != AR_SUCCESS) { + if (shouldLog) { + lastLogNs = nowNs; + info("ARCore ERROR: ArSession_update failed (status %d)", (int) status); + } + return; + } + + ArCamera *arCamera = nullptr; + ArFrame_acquireCamera(CTX.arSession, CTX.arFrame, &arCamera); + + ArTrackingState trackingState; + ArCamera_getTrackingState(CTX.arSession, arCamera, &trackingState); + + if (trackingState != AR_TRACKING_STATE_TRACKING) { + // Not tracking: keep the last known position and correction. + if (shouldLog) { + lastLogNs = nowNs; + info("ARCore: not tracking (state %d)", (int) trackingState); + } + ArCamera_release(arCamera); + return; + } + + ArPose *arPose = nullptr; + ArPose_create(CTX.arSession, nullptr, &arPose); + ArCamera_getDisplayOrientedPose(CTX.arSession, arCamera, arPose); + // {qx, qy, qz, qw, tx, ty, tz} + float raw[7] = {}; + ArPose_getPoseRaw(CTX.arSession, arPose, raw); + ArPose_destroy(arPose); + ArCamera_release(arCamera); + + // Vertical position (see ARCORE_HEIGHT_MODE; the mode in use is CTX.heightMode). + float headY = raw[5]; + float floorY = 0.0f; + bool floorKnown = false; + if (CTX.heightMode != 0) { + floorKnown = updateFloorAnchor(&floorY); + } + if (CTX.heightMode == 1) { + if (floorKnown) { + headY = raw[5] - floorY; + } + } else if (CTX.heightMode == 2) { + if (floorKnown) { + uint64_t blendDtNs = CTX.lastFloorBlendNs != 0 ? nowNs - CTX.lastFloorBlendNs : 0; + if (blendDtNs > 0 && blendDtNs < ARCORE_MAX_CORRECTION_GAP_NS) { + float gain = 1.0f - std::exp(-((float) blendDtNs * 1e-9f) / + ARCORE_FLOOR_BLEND_TIME_CONSTANT_S); + // ARCore's origin is -floorY above the floor. + CTX.floorOffsetM += (-floorY - CTX.floorOffsetM) * gain; + } + CTX.lastFloorBlendNs = nowNs; + } else { + CTX.lastFloorBlendNs = 0; + } + headY = raw[5] + CTX.floorOffsetM; + } + headY += ARCORE_HEIGHT_OFFSET_M; + + // How far apart are Cardboard and ARCore right now? + AlvrQuat cardboardOrientation = + cardboardOrientationAt(nowNs + ESTIMATED_PREDICTION_OFFSET_NS); + AlvrQuat arcoreOrientation = normalizeQuat(AlvrQuat{raw[0], raw[1], raw[2], raw[3]}); + AlvrQuat arcoreInCardboard = + normalizeQuat(quatMultiply(arcoreOrientation, quatInverse(ARCORE_TO_CARDBOARD))); + + // The rotation, in world space, that takes Cardboard's orientation onto ARCore's. Both sensors + // know which way is down, so this is (nearly) a pure rotation about the vertical: the yaw + // drift. Splitting off that yaw (and treating what is left as "tilt") makes the correction + // independent of where the head is pointing. A correction expressed in the phone's own frame + // is only right while the head keeps the pose it was measured in. + AlvrQuat worldOffset = + normalizeQuat(quatMultiply(arcoreInCardboard, quatInverse(cardboardOrientation))); + if (worldOffset.w < 0.0f) { + worldOffset = {-worldOffset.x, -worldOffset.y, -worldOffset.z, -worldOffset.w}; + } + const float offsetComponents[3] = {worldOffset.x, worldOffset.y, worldOffset.z}; + float measuredYawRad = + 2.0f * std::atan2(offsetComponents[ARCORE_WORLD_UP_AXIS], worldOffset.w); + AlvrQuat tilt = normalizeQuat( + quatMultiply(quatInverse(worldYawQuat(measuredYawRad)), worldOffset)); + float tiltDeg = quatAngleDegrees(tilt); + float yawOffsetDeg = measuredYawRad * 180.0f / (float) M_PI; + + static int acceptedCount = 0; + static int rejectedCount = 0; + static float maxResidualDeg = 0.0f; + static float maxTiltDeg = 0.0f; + static bool wasReacquiring = false; + + float residualDeg = 0.0f; + bool accepted = false; + bool reacquiring = false; + float correctionDeg = 0.0f; + + { + std::lock_guard poseLock(CTX.poseMutex); + + CTX.arPosition[0] = raw[4]; + CTX.arPosition[1] = headY; + CTX.arPosition[2] = raw[6]; + CTX.arPositionValid = true; + + // How much ARCore disagrees with the yaw being applied right now. + float residualRad = wrapRad(measuredYawRad - CTX.yawCorrectionRad); + residualDeg = std::abs(residualRad) * 180.0f / (float) M_PI; + + bool agrees = residualDeg < ARCORE_MAX_RESIDUAL_DEG; + if (agrees) { + CTX.disagreeSinceNs = 0; + } else if (CTX.disagreeSinceNs == 0) { + CTX.disagreeSinceNs = nowNs; + } + uint64_t reacquireAfterNs = (uint64_t) (ARCORE_REACQUIRE_AFTER_S * 1e9f); + reacquiring = !agrees && (nowNs - CTX.disagreeSinceNs) >= reacquireAfterNs; + accepted = agrees || reacquiring; + + if (accepted) { + if (CTX.lastCorrectionNs == 0) { + CTX.lastCorrectionNs = nowNs; + } + uint64_t deltaNs = nowNs - CTX.lastCorrectionNs; + // Protect against an unexpectedly large gap (pause, debugger, lifecycle change). + if (deltaNs > 0 && deltaNs < ARCORE_MAX_CORRECTION_GAP_NS) { + float dt = (float) deltaNs * 1e-9f; + float timeConstant = + agrees ? ARCORE_CORRECTION_TIME_CONSTANT_S : ARCORE_REACQUIRE_TIME_CONSTANT_S; + // Time-based gain, so the result doesn't depend on the frame rate. + float gain = 1.0f - std::exp(-dt / timeConstant); + CTX.yawCorrectionRad = wrapRad(CTX.yawCorrectionRad + residualRad * gain); + } + CTX.lastCorrectionNs = nowNs; + } else { + // Don't let a transient mismatch pull the correction toward a bad measurement. + CTX.lastCorrectionNs = nowNs; + } + + correctionDeg = CTX.yawCorrectionRad * 180.0f / (float) M_PI; + } - AlvrPose headPose = getPose(targetTimestampNs); + if (accepted) { + acceptedCount++; + } else { + rejectedCount++; + } + maxResidualDeg = std::max(maxResidualDeg, residualDeg); + maxTiltDeg = std::max(maxTiltDeg, tiltDeg); + if (reacquiring && !wasReacquiring) { + info("ARCore: yaw disagreed for %.1f s (residual %.1f deg, offset %.1f deg); following " + "ARCore's new yaw", + ARCORE_REACQUIRE_AFTER_S, residualDeg, yawOffsetDeg); + } + wasReacquiring = reacquiring; + + if (shouldLog) { + lastLogNs = nowNs; + info("ARCore: tracking, position=(%.3f, %.3f, %.3f), floor %s (y=%.2f) | yaw offset from " + "ARCore=%.2f deg, residual=%.2f deg (max %.2f), yaw correction=%.2f deg, tilt " + "disagreement=%.2f deg (max %.2f), last second: %d accepted, %d rejected%s", + raw[4], headY, raw[6], floorKnown ? "known" : "unknown", floorY, yawOffsetDeg, + residualDeg, maxResidualDeg, correctionDeg, tiltDeg, maxTiltDeg, acceptedCount, + rejectedCount, reacquiring ? ", RE-ACQUIRING" : ""); + acceptedCount = 0; + rejectedCount = 0; + maxResidualDeg = 0.0f; + maxTiltDeg = 0.0f; + } +} - CTX.deviceMotion.device_id = HEAD_ID; - CTX.deviceMotion.pose = headPose; +// Local-space (head-at-origin) per-eye view parameters: just the eye offset and FOV. These rarely +// change (only when the lens distortion / viewer profile is recalculated), so this is sent once +// via alvr_send_view_params() rather than on every tracking update - unlike the v20 API, the +// moving head pose is no longer bundled into this call. +void sendViewParams() { + AlvrViewParams viewParams[2] = {}; - float headToEye[3] = {CTX.eyeOffsets[kLeft], 0.0, 0.0}; + for (int eye = 0; eye < 2; eye++) { + viewParams[eye].pose.orientation = AlvrQuat{0.0, 0.0, 0.0, 1.0}; + viewParams[eye].pose.position[0] = CTX.eyeOffsets[eye]; + viewParams[eye].pose.position[1] = 0.0; + viewParams[eye].pose.position[2] = 0.0; + viewParams[eye].fov = CTX.fovArr[eye]; + } - CTX.viewParams[kLeft].pose = headPose; - offsetPosWithQuat(headPose.orientation, headToEye, CTX.viewParams[kLeft].pose.position); - CTX.viewParams[kLeft].fov = CTX.fovArr[kLeft]; + alvr_send_view_params(viewParams); +} - headToEye[0] = CTX.eyeOffsets[kRight]; - CTX.viewParams[kRight].pose = headPose; - offsetPosWithQuat(headPose.orientation, headToEye, CTX.viewParams[kRight].pose.position); - CTX.viewParams[kRight].fov = CTX.fovArr[kRight]; +// Once a second: how the own head filter compares with Cardboard's tracker. "tilt diff" should +// stay under a degree or two (a big value means the sensor axes are mapped wrongly). "yaw diff" +// drifts slowly (that is the drift being removed); turn your head left and right: if it jumps +// around instead of staying put, the yaw direction is mirrored. +static void logHeadFilterCheck(uint64_t nowNs) { + static uint64_t lastNs = 0; + if (!USE_OWN_HEAD_FILTER || !HEAD_FILTER_DEBUG_LOG || !CTX.headFilter.ready()) { + return; + } + if (nowNs - lastNs < 1000000000ULL) { + return; + } + lastNs = nowNs; + AlvrQuat own = cardboardOrientationAt(nowNs); + AlvrQuat cardboard = cardboardTrackerOrientationAt(nowNs); + float yawDiffDeg = wrapRad(facingYawRad(own) - facingYawRad(cardboard)) * 180.f / (float) M_PI; + phonevr::HeadOrientationFilter::Diagnostics d = CTX.headFilter.diagnostics(); + info("HeadFilter: tilt diff vs Cardboard %.2f deg, yaw diff %.1f deg | gyro bias (deg/s) " + "%.3f %.3f %.3f%s", + upVectorAngleDeg(own, cardboard), yawDiffDeg, d.biasDps[0], d.biasDps[1], d.biasDps[2], + d.biasReady ? "" : " (not measured yet: hold still)"); } void inputThread() { @@ -144,12 +815,13 @@ void inputThread() { info("inputThread: thread staring..."); while (CTX.streaming) { + auto targetTimestampNs = GetBootTimeNano() + ESTIMATED_PREDICTION_OFFSET_NS; + logHeadFilterCheck(targetTimestampNs); - auto targetTimestampNs = GetBootTimeNano() + alvr_get_head_prediction_offset_ns(); - updateViewConfigs(targetTimestampNs); + CTX.deviceMotion.device_id = HEAD_ID; + CTX.deviceMotion.pose = getPose(targetTimestampNs); - alvr_send_tracking( - targetTimestampNs, CTX.viewParams, &CTX.deviceMotion, 1, nullptr, nullptr); + alvr_send_tracking(targetTimestampNs, &CTX.deviceMotion, 1, nullptr, nullptr); deadline += std::chrono::nanoseconds((uint64_t) (1e9 / 60.f / 3)); std::this_thread::sleep_until(deadline); @@ -162,32 +834,95 @@ extern "C" JNIEXPORT jint JNI_OnLoad(JavaVM *vm, void *) { } extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_initializeNative( - JNIEnv *env, jobject obj, jint screenWidth, jint screenHeight, jfloat refreshRate) { + JNIEnv *env, jobject obj, jint screenWidth, jint screenHeight, jfloat refreshRate,jboolean enableARCore) { CTX.javaContext = env->NewGlobalRef(obj); + info("initializeNative: enableARCore=%d", (int) enableARCore); - uint32_t viewWidth = std::max(screenWidth, screenHeight) / 2; - uint32_t viewHeight = std::min(screenWidth, screenHeight); + uint32_t viewWidth = (std::max(screenWidth, screenHeight) / 2) & ~15u; // multiple of 16 + uint32_t viewHeight = std::min(screenWidth, screenHeight) & ~7u; // multiple of 8 + + // Route the Rust library's log output (including the reason for its aborts) to logcat. + alvr_initialize_logging(); alvr_initialize_android_context((void *) CTX.javaVm, (void *) CTX.javaContext); - float refreshRatesBuffer[1] = {refreshRate}; + static float refreshRatesBuffer[1]; + refreshRatesBuffer[0] = refreshRate; AlvrClientCapabilities caps = {}; - caps.default_view_height = viewHeight; caps.default_view_width = viewWidth; - caps.external_decoder = false; + caps.default_view_height = viewHeight; + caps.max_view_width = viewWidth; + caps.max_view_height = viewHeight; caps.refresh_rates = refreshRatesBuffer; caps.refresh_rates_count = 1; - caps.foveated_encoding = - true; // By default disable FFE (can be force-enabled by Server Settings + caps.foveated_encoding = false; caps.encoder_high_profile = true; - caps.encoder_10_bits = true; - caps.encoder_av1 = true; + caps.encoder_10_bits = false; + caps.encoder_av1 = false; + caps.prefer_10bit = false; + caps.preferred_encoding_gamma = 1.0; + caps.prefer_hdr = false; alvr_initialize(caps); Cardboard_initializeAndroid(CTX.javaVm, CTX.javaContext); CTX.headTracker = CardboardHeadTracker_create(); + + CTX.arcoreEnabled = (bool) enableARCore; + if (CTX.arcoreEnabled) { + if (ArSession_create(env, CTX.javaContext, &CTX.arSession) != AR_SUCCESS) { + info("ARCore ERROR: could not create session (is Google Play Services for AR " + "installed?); continuing without ARCore"); + CTX.arSession = nullptr; + CTX.arcoreEnabled = false; + } + } + + if (CTX.arcoreEnabled) { + ArConfig *arConfig = nullptr; + ArConfig_create(CTX.arSession, &arConfig); + + // Explicitly disable everything we don't need, to preserve CPU and battery. + ArConfig_setDepthMode(CTX.arSession, arConfig, AR_DEPTH_MODE_DISABLED); + ArConfig_setLightEstimationMode( + CTX.arSession, arConfig, AR_LIGHT_ESTIMATION_MODE_DISABLED); + ArConfig_setPlaneFindingMode( + CTX.arSession, arConfig, AR_PLANE_FINDING_MODE_HORIZONTAL_AND_VERTICAL); + ArConfig_setCloudAnchorMode(CTX.arSession, arConfig, AR_CLOUD_ANCHOR_MODE_DISABLED); + // Don't block waiting for a new camera image. + ArConfig_setUpdateMode(CTX.arSession, arConfig, AR_UPDATE_MODE_LATEST_CAMERA_IMAGE); + + ArStatus configureStatus = ArSession_configure(CTX.arSession, arConfig); + ArConfig_destroy(arConfig); + + if (configureStatus != AR_SUCCESS) { + info("ARCore ERROR: could not configure session (status %d); continuing without " + "ARCore", + (int) configureStatus); + ArSession_destroy(CTX.arSession); + CTX.arSession = nullptr; + CTX.arcoreEnabled = false; + } else { + ArFrame_create(CTX.arSession, &CTX.arFrame); + info("ARCore session created and configured"); + } + } +} + +// Settings from the app's Settings screen. Called right after initializeNative(), before the +// stream starts. +extern "C" JNIEXPORT void JNICALL +Java_viritualisres_phonevr_ALVRActivity_setStreamFovScaleNative(JNIEnv *, jobject, jfloat scale) { + // Keep it in a sane range whatever was stored. + CTX.streamFovScale = std::min(1.0f, std::max(0.5f, (float) scale)); + info("Setting: stream FOV scale %.2f", CTX.streamFovScale); +} + +extern "C" JNIEXPORT void JNICALL +Java_viritualisres_phonevr_ALVRActivity_setFloorModeNative(JNIEnv *, jobject, jint mode) { + CTX.heightMode = (mode >= 0 && mode <= 2) ? (int) mode : ARCORE_HEIGHT_MODE; + info("Setting: ARCore floor/height mode %d", CTX.heightMode); } extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_destroyNative(JNIEnv *, @@ -195,17 +930,42 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_destro alvr_destroy_opengl(); alvr_destroy(); + CTX.headSensors.stop(); CardboardHeadTracker_destroy(CTX.headTracker); CTX.headTracker = nullptr; CardboardLensDistortion_destroy(CTX.lensDistortion); CTX.lensDistortion = nullptr; CardboardDistortionRenderer_destroy(CTX.distortionRenderer); CTX.distortionRenderer = nullptr; + + { + std::lock_guard lock(CTX.arMutex); + if (CTX.arFloorAnchor != nullptr) { + ArAnchor_release(CTX.arFloorAnchor); + CTX.arFloorAnchor = nullptr; + } + if (CTX.arFrame != nullptr) { + ArFrame_destroy(CTX.arFrame); + CTX.arFrame = nullptr; + } + if (CTX.arSession != nullptr) { + ArSession_destroy(CTX.arSession); + CTX.arSession = nullptr; + } + CTX.arcoreEnabled = false; + } } extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_resumeNative(JNIEnv *, jobject) { CardboardHeadTracker_resume(CTX.headTracker); + if (USE_OWN_HEAD_FILTER) { + CTX.headSensors.start(&CTX.headFilter); + } + + if (!ARCORE_START_AFTER_VIDEO) { + startARCoreSession("app resumed"); + } CTX.renderingParamsChanged = true; @@ -218,8 +978,11 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_resume CardboardQrCode_destroy(buffer); CTX.running = true; - alvr_resume(); + + // at the start of renderNative, right after alvr_resume(...), and right before the final draw: + LOG("EGLCHK ctx=%p surf=%p", eglGetCurrentContext(), eglGetCurrentSurface(EGL_DRAW)); + EGLCHK("after alvr_resume()"); } extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_pauseNative(JNIEnv *, @@ -231,11 +994,16 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_pauseN } CardboardHeadTracker_pause(CTX.headTracker); + CTX.headSensors.stop(); + + // Releases the camera. (The v20 code never paused the ARCore session.) + stopARCoreSession("app paused"); } extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_surfaceCreatedNative(JNIEnv *, jobject) { - alvr_initialize_opengl(); + logGlInfo(); + ALVR_GL_CALL(alvr_initialize_opengl()); CTX.glContextRecreated = true; } @@ -255,73 +1023,216 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_sendBa extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_renderNative(JNIEnv *, jobject) { - try { - if (CTX.renderingParamsChanged) { - info("renderingParamsChanged, processing new params"); - uint8_t *buffer; - int size; - CardboardQrCode_getSavedDeviceParams(&buffer, &size); - - if (size == 0) { - return; - } + // at the start of renderNative, right after alvr_resume(...), and right before the final draw: + LOG("EGLCHK ctx=%p surf=%p", eglGetCurrentContext(), eglGetCurrentSurface(EGL_DRAW)); + EGLCHK("Start of renderNative()"); + + if (CTX.renderingParamsChanged) { + info("renderingParamsChanged, processing new params"); + uint8_t *buffer; + int size; + CardboardQrCode_getSavedDeviceParams(&buffer, &size); + + if (size == 0) { + return; + } - info("renderingParamsChanged, sending new params to alvr"); - if (CTX.lensDistortion) { - CardboardLensDistortion_destroy(CTX.lensDistortion); - CTX.lensDistortion = nullptr; - } - info("renderingParamsChanged, destroyed distortion"); - CTX.lensDistortion = - CardboardLensDistortion_create(buffer, size, CTX.screenWidth, CTX.screenHeight); + info("renderingParamsChanged, sending new params to alvr"); + if (CTX.lensDistortion) { + CardboardLensDistortion_destroy(CTX.lensDistortion); + CTX.lensDistortion = nullptr; + } + info("renderingParamsChanged, destroyed distortion"); + CTX.lensDistortion = + CardboardLensDistortion_create(buffer, size, CTX.screenWidth, CTX.screenHeight); - CardboardQrCode_destroy(buffer); - *buffer = 0; + CardboardQrCode_destroy(buffer); + *buffer = 0; - if (CTX.distortionRenderer) { - CardboardDistortionRenderer_destroy(CTX.distortionRenderer); - CTX.distortionRenderer = nullptr; - } - const CardboardOpenGlEsDistortionRendererConfig config{kGlTexture2D}; - CTX.distortionRenderer = CardboardOpenGlEs2DistortionRenderer_create(&config); - - for (int eye = 0; eye < 2; eye++) { - CardboardMesh mesh; - CardboardLensDistortion_getDistortionMesh( - CTX.lensDistortion, (CardboardEye) eye, &mesh); - CardboardDistortionRenderer_setMesh( - CTX.distortionRenderer, &mesh, (CardboardEye) eye); - - float matrix[16] = {}; - CardboardLensDistortion_getEyeFromHeadMatrix( - CTX.lensDistortion, (CardboardEye) eye, matrix); - CTX.eyeOffsets[eye] = matrix[12]; - } + if (CTX.distortionRenderer) { + CardboardDistortionRenderer_destroy(CTX.distortionRenderer); + CTX.distortionRenderer = nullptr; + } + const CardboardOpenGlEsDistortionRendererConfig config{kGlTexture2D}; + CTX.distortionRenderer = CardboardOpenGlEs2DistortionRenderer_create(&config); + + for (int eye = 0; eye < 2; eye++) { + CardboardMesh mesh; + CardboardLensDistortion_getDistortionMesh( + CTX.lensDistortion, (CardboardEye) eye, &mesh); + CardboardDistortionRenderer_setMesh( + CTX.distortionRenderer, &mesh, (CardboardEye) eye); + + float matrix[16] = {}; + CardboardLensDistortion_getEyeFromHeadMatrix( + CTX.lensDistortion, (CardboardEye) eye, matrix); + CTX.eyeOffsets[eye] = matrix[12]; + } - CTX.fovArr[kLeft] = getFov(kLeft); - CTX.fovArr[kRight] = getFov(kRight); + CTX.fovArr[kLeft] = getStreamFov(kLeft); + CTX.fovArr[kRight] = getStreamFov(kRight); - info("renderingParamsChanged, updating new view configs (FOV) to alvr"); - // alvr_send_views_config(fovArr, CTX.eyeOffsets[0] - CTX.eyeOffsets[1]); - } + info("renderingParamsChanged, sending new view configs (FOV) to alvr"); + sendViewParams(); + } + + // Note: if GL context is recreated, old resources are already freed. + if (CTX.renderingParamsChanged && !CTX.glContextRecreated) { + info("Pausing ALVR since glContext is not recreated, deleting textures"); + ALVR_GL_CALL(alvr_pause_opengl()); + + GL(glDeleteTextures(2, CTX.lobbyTextures)); + } - // Note: if GL context is recreated, old resources are already freed. - if (CTX.renderingParamsChanged && !CTX.glContextRecreated) { - info("Pausing ALVR since glContext is not recreated, deleting textures"); - alvr_pause_opengl(); + if (CTX.renderingParamsChanged || CTX.glContextRecreated) { + info("Rebuilding, binding textures, Resuming ALVR since glContextRecreated %b, " + "renderingParamsChanged %b", + CTX.renderingParamsChanged, + CTX.glContextRecreated); +#ifdef PVR_TEXTURE_NAME_TEST + { + static bool burned = false; + if (!burned) { + GLuint burn[16]; + glGenTextures(16, burn); + burned = true; + } + } +#endif + GL(glGenTextures(2, CTX.lobbyTextures)); + + for (auto &lobbyTexture : CTX.lobbyTextures) { + GL(glBindTexture(GL_TEXTURE_2D, lobbyTexture)); + GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE)); + GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE)); + GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR)); + GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR)); + GL(glTexImage2D(GL_TEXTURE_2D, + 0, + GL_RGB, + CTX.screenWidth / 2, + CTX.screenHeight, + 0, + GL_RGB, + GL_UNSIGNED_BYTE, + nullptr)); + } - GL(glDeleteTextures(2, CTX.lobbyTextures)); + // The ALVR library renders in its own EGL context (now a share context of this one). + // Object creation/changes in this context are only guaranteed visible to the other + // context after a flush/finish, so finish before handing the texture names over. + glFinish(); + LOG("LOBBYTEX names=%u,%u isTexture=%d,%d size=%dx%d", + (unsigned) CTX.lobbyTextures[0], + (unsigned) CTX.lobbyTextures[1], + (int) glIsTexture(CTX.lobbyTextures[0]), + (int) glIsTexture(CTX.lobbyTextures[1]), + CTX.screenWidth / 2, + CTX.screenHeight); + + const uint32_t *targetViews[2] = {(const uint32_t *) &CTX.lobbyTextures[0], + (const uint32_t *) &CTX.lobbyTextures[1]}; + // v21 dropped the trailing bool (was "always direct") - resume takes 4 args now. + // alvr_resume_opengl() leaves the thread with no current EGL context (observed: + // ctx/surf go to 0x0), which makes every later GL call fail and GLSurfaceView stop + // rendering. The guard captures the app's EGL state before the call and rebinds it + // immediately afterwards. + EGLCHK("before alvr_resume_opengl"); + { + EglStateGuard guard; + alvr_resume_opengl(CTX.screenWidth / 2, CTX.screenHeight, targetViews, 1); + } + EGLCHK("after alvr_resume_opengl (restored)"); + + // ARCore needs a GL texture to write camera frames into. We never draw the camera + // image, but ARCore requires one. If the GL context was recreated the old texture is + // already gone; otherwise delete it so it doesn't leak. + if (CTX.arcoreEnabled && CTX.arSession != nullptr) { + if (CTX.arTexture != 0 && !CTX.glContextRecreated) { + glDeleteTextures(1, &CTX.arTexture); + } + GLuint arTextureIds[1] = {0}; + glGenTextures(1, arTextureIds); + GL(glBindTexture(GL_TEXTURE_2D, arTextureIds[0])); + GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE)); + GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE)); + GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR)); + GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR)); + + { + std::lock_guard lock(CTX.arMutex); + CTX.arTexture = arTextureIds[0]; + ArSession_setCameraTextureName(CTX.arSession, CTX.arTexture); + } + info("ARCore camera texture initialized: %u", (unsigned) CTX.arTexture); } - if (CTX.renderingParamsChanged || CTX.glContextRecreated) { - info("Rebuilding, binding textures, Resuming ALVR since glContextRecreated %b, " - "renderingParamsChanged %b", - CTX.renderingParamsChanged, - CTX.glContextRecreated); - GL(glGenTextures(2, CTX.lobbyTextures)); + CTX.renderingParamsChanged = false; + CTX.glContextRecreated = false; + } + + AlvrEvent event; + while (alvr_poll_event(&event)) { + if (event.tag == ALVR_EVENT_HUD_MESSAGE_UPDATED) { + auto message_length = alvr_hud_message(nullptr); + auto message_buffer = std::vector(message_length); - for (auto &lobbyTexture : CTX.lobbyTextures) { - GL(glBindTexture(GL_TEXTURE_2D, lobbyTexture)); + alvr_hud_message(&message_buffer[0]); + info("ALVR Poll Event: HUD Message Update - %s", &message_buffer[0]); + + // 3) inside the HUD branch, right after the call + if (message_length > 0) + ALVR_GL_CALL(alvr_update_hud_message_opengl(&message_buffer[0])); + EGLCHK("after update_hud_message"); + } + if (event.tag == ALVR_EVENT_STREAMING_STARTED) { + info("ALVR Poll Event: ALVR_EVENT_STREAMING_STARTED, generating and binding " + "textures..."); + auto &started = event.STREAMING_STARTED; + + info("Streaming started: %dx%d, refresh hint=%f, gamma=%f, hdr=%b, ffe=%b", + started.view_width, + started.view_height, + started.refresh_rate_hint, + started.encoding_gamma, + started.enable_hdr, + started.enable_foveated_encoding); + + // v21 delivers the negotiated foveation params directly in the event - no more + // manual settings-JSON parsing needed here (settings.video.foveated_encoding, as + // the v20 code did). + CTX.hasFrameMetadata = false; + CTX.lastStreamBuffer = nullptr; + CTX.videoFramesSeen = 0; + + // Upscaling: the stream is started_view_width x height, but when upscaling is on ALVR + // renders into textures that are upscale_factor times bigger (it truncates the + // product to whole pixels, exactly as done here), so the textures we hand over have + // to be that big too. A texture of the wrong size is what turned the screen black. + float upscaleFactor = 1.0f; + if (ENABLE_UPSCALING) { + float wantW = (float) (CTX.screenWidth / 2) / (float) started.view_width; + float wantH = (float) CTX.screenHeight / (float) started.view_height; + float want = std::min(wantW, wantH); + if (want >= 1.05f) { // stream is clearly smaller than the display + upscaleFactor = std::min(want, UPSCALING_MAX_FACTOR); + } + } + const bool upscaling = upscaleFactor > 1.0f; + const uint32_t targetWidth = (uint32_t) ((float) started.view_width * upscaleFactor); + const uint32_t targetHeight = (uint32_t) ((float) started.view_height * upscaleFactor); + info("Upscaling %s: stream %dx%d, factor %.3f, textures %ux%u", + upscaling ? "ON" : "off", + started.view_width, + started.view_height, + upscaleFactor, + targetWidth, + targetHeight); + + GL(glGenTextures(2, CTX.streamTextures)); + + for (auto &streamTexture : CTX.streamTextures) { + GL(glBindTexture(GL_TEXTURE_2D, streamTexture)); GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE)); GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE)); GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR)); @@ -329,212 +1240,249 @@ extern "C" JNIEXPORT void JNICALL Java_viritualisres_phonevr_ALVRActivity_render GL(glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, - CTX.screenWidth / 2, - CTX.screenHeight, + targetWidth, + targetHeight, 0, GL_RGB, GL_UNSIGNED_BYTE, nullptr)); } - const uint32_t *targetViews[2] = {(uint32_t *) &CTX.lobbyTextures[0], - (uint32_t *) &CTX.lobbyTextures[1]}; - alvr_resume_opengl(CTX.screenWidth / 2, CTX.screenHeight, targetViews, 1, true); - - CTX.renderingParamsChanged = false; - CTX.glContextRecreated = false; - } + CTX.fovArr[0] = getStreamFov((CardboardEye) 0); + CTX.fovArr[1] = getStreamFov((CardboardEye) 1); + sendViewParams(); + + info("ALVR Poll Event: ALVR_EVENT_STREAMING_STARTED, View configs updated..."); + + auto leftIntHandle = (uint32_t) CTX.streamTextures[0]; + auto rightIntHandle = (uint32_t) CTX.streamTextures[1]; + const uint32_t *textureHandles[2] = {&leftIntHandle, &rightIntHandle}; + + auto render_config = AlvrStreamConfig{}; + render_config.view_resolution_width = started.view_width; + render_config.view_resolution_height = started.view_height; + render_config.swapchain_textures = textureHandles; + render_config.swapchain_length = 1; + + render_config.enable_foveation = started.enable_foveated_encoding; + render_config.foveated_encoding = started.foveated_encoding; + + // view_resolution above stays the size of the decoded stream; the textures are the + // upscaled size (see above). + render_config.enable_upscaling = upscaling; + if (upscaling) { + render_config.upscaling_edge_direction = UPSCALING_EDGE_DIRECTION; + render_config.upscaling_edge_threshold = UPSCALING_EDGE_THRESHOLD; + render_config.upscaling_edge_sharpness = UPSCALING_EDGE_SHARPNESS; + render_config.upscale_factor = upscaleFactor; + } - AlvrEvent event; - while (alvr_poll_event(&event)) { - if (event.tag == ALVR_EVENT_HUD_MESSAGE_UPDATED) { - auto message_length = alvr_hud_message(nullptr); - auto message_buffer = std::vector(message_length); + ALVR_GL_CALL(alvr_start_stream_opengl(render_config)); + + info("ALVR Poll Event: ALVR_EVENT_STREAMING_STARTED, opengl stream started and " + "input Thread started..."); + CTX.streaming = true; + CTX.inputThread = std::thread(inputThread); + } else if (event.tag == ALVR_EVENT_DECODER_CONFIG) { + + /* + * v21 no longer sends the decoder config in the ALVR_EVENT_STREAMING_STARTED event, + * so we have to create the decoder here instead. The decoder config is sent in a + * separate event, ALVR_EVENT_DECODER_CONFIG, which is guaranteed to arrive before + * the first frame. + */ + if (!g_decoderCreated) { + auto configSize = alvr_get_decoder_config(nullptr); + auto configBuffer = std::vector(configSize); + alvr_get_decoder_config(configBuffer.data()); + + // false = use the phone's hardware decoder. true forces ALVR's software decoder, + // which cannot be created on some phones (creating it then hangs), so leave it false. + constexpr bool FORCE_SOFTWARE_DECODER = false; + + // Same options the stock ALVR client gets from the server's defaults. + AlvrMediacodecOption decoderOptions[3] = {}; + decoderOptions[0].key = "operating-rate"; + decoderOptions[0].ty = ALVR_MEDIACODEC_PROP_TYPE_INT32; + decoderOptions[0].value.int32 = INT32_MAX; + decoderOptions[1].key = "priority"; + decoderOptions[1].ty = ALVR_MEDIACODEC_PROP_TYPE_INT32; + decoderOptions[1].value.int32 = 0; + decoderOptions[2].key = "vendor.qti-ext-dec-low-latency.enable"; + decoderOptions[2].ty = ALVR_MEDIACODEC_PROP_TYPE_INT32; + decoderOptions[2].value.int32 = 1; + + AlvrDecoderConfig decoderConfig = {}; + decoderConfig.codec = event.DECODER_CONFIG.codec; + decoderConfig.force_software_decoder = FORCE_SOFTWARE_DECODER; + decoderConfig.max_buffering_frames = 1.5f; + decoderConfig.buffering_history_weight = 0.9f; + decoderConfig.options = decoderOptions; + decoderConfig.options_count = 3; + decoderConfig.config_buffer = + reinterpret_cast(configBuffer.data()); + decoderConfig.config_buffer_size = configSize; + + info("ALVR Poll Event: ALVR_EVENT_DECODER_CONFIG, codec=%d, config bytes=%llu, " + "software=%d", + (int) decoderConfig.codec, (unsigned long long) configSize, + (int) FORCE_SOFTWARE_DECODER); + // The saved frame belongs to the previous decoder, if there was one. + CTX.lastStreamBuffer = nullptr; + alvr_create_decoder(decoderConfig); + g_decoderCreated = true; + } - alvr_hud_message(&message_buffer[0]); - info("ALVR Poll Event: HUD Message Update - %s", &message_buffer[0]); + } else if (event.tag == ALVR_EVENT_STREAMING_STOPPED) { + info("ALVR Poll Event: ALVR_EVENT_STREAMING_STOPPED, Waiting for inputThread to " + "join..."); + CTX.streaming = false; + CTX.inputThread.join(); - if (message_length > 0) - alvr_update_hud_message_opengl(&message_buffer[0]); + // The camera must be off before the next decoder is created. + if (ARCORE_START_AFTER_VIDEO) { + stopARCoreSession("stream stopped"); } - if (event.tag == ALVR_EVENT_STREAMING_STARTED) { - info("ALVR Poll Event: ALVR_EVENT_STREAMING_STARTED, generating and binding " - "textures..."); - auto config = event.STREAMING_STARTED; - - auto settings_len = alvr_get_settings_json(nullptr); - auto settings_buffer = std::vector(settings_len); - alvr_get_settings_json(&settings_buffer[0]); - - info("Got settings from ALVR Server - %s", &settings_buffer[0]); - if (settings_len > 900) // to workthough logcat buffer limit - info("Got settings from ALVR Server - %s", &settings_buffer[900]); - json settings_json = json::parse(&settings_buffer[0]); - - GL(glGenTextures(2, CTX.streamTextures)); - - for (auto &streamTexture : CTX.streamTextures) { - GL(glBindTexture(GL_TEXTURE_2D, streamTexture)); - GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE)); - GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE)); - GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR)); - GL(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR)); - GL(glTexImage2D(GL_TEXTURE_2D, - 0, - GL_RGB, - config.view_width, - config.view_height, - 0, - GL_RGB, - GL_UNSIGNED_BYTE, - nullptr)); - } - CTX.fovArr[0] = getFov((CardboardEye) 0); - CTX.fovArr[1] = getFov((CardboardEye) 1); - - info("ALVR Poll Event: ALVR_EVENT_STREAMING_STARTED, View configs updated..."); - - auto leftIntHandle = (uint32_t) CTX.streamTextures[0]; - auto rightIntHandle = (uint32_t) CTX.streamTextures[1]; - const uint32_t *textureHandles[2] = {&leftIntHandle, &rightIntHandle}; - - auto render_config = AlvrStreamConfig{}; - render_config.view_resolution_width = config.view_width; - render_config.view_resolution_height = config.view_height; - render_config.swapchain_textures = textureHandles; - render_config.swapchain_length = 1; - - render_config.enable_foveation = false; - if (!settings_json["video"].is_null()) { - if (!settings_json["video"]["foveated_encoding"].is_null()) { - info("settings_json.video.foveated_encoding is %s", - settings_json["video"]["foveated_encoding"].dump().c_str()); - - // Foveated encoding would be a "Enabled": {Array} or "Disabled" String - if (!settings_json["video"]["foveated_encoding"].is_string()) { - render_config.enable_foveation = true; - render_config.foveation_center_size_x = - settings_json["video"]["foveated_encoding"]["Enabled"] - ["center_size_x"]; - render_config.foveation_center_size_y = - settings_json["video"]["foveated_encoding"]["Enabled"] - ["center_size_y"]; - render_config.foveation_center_shift_x = - settings_json["video"]["foveated_encoding"]["Enabled"] - ["center_shift_x"]; - render_config.foveation_center_shift_y = - settings_json["video"]["foveated_encoding"]["Enabled"] - ["center_shift_y"]; - render_config.foveation_edge_ratio_x = - settings_json["video"]["foveated_encoding"]["Enabled"] - ["edge_ratio_x"]; - render_config.foveation_edge_ratio_y = - settings_json["video"]["foveated_encoding"]["Enabled"] - ["edge_ratio_y"]; - } else - info("foveated_encoding is Disabled"); - } else - error("settings_json doesn't have a video.foveated_encoding key"); - } else - error("settings_json doesn't have a video key"); - - info("Settings for foveation:"); - info("render_config.enable_foveation: %b", render_config.enable_foveation); - info("render_config.foveation_center_size_x: %f", - render_config.foveation_center_size_x); - info("render_config.foveation_center_size_y: %f", - render_config.foveation_center_size_y); - info("render_config.foveation_center_shift_x: %f", - render_config.foveation_center_shift_x); - info("render_config.foveation_center_shift_y: %f", - render_config.foveation_center_shift_y); - info("render_config.foveation_edge_ratio_x: %f", - render_config.foveation_edge_ratio_x); - info("render_config.foveation_edge_ratio_y: %f", - render_config.foveation_edge_ratio_y); - - alvr_start_stream_opengl(render_config); - - info("ALVR Poll Event: ALVR_EVENT_STREAMING_STARTED, opengl stream started and " - "input " - "Thread started..."); - CTX.streaming = true; - CTX.inputThread = std::thread(inputThread); - - } else if (event.tag == ALVR_EVENT_STREAMING_STOPPED) { - info("ALVR Poll Event: ALVR_EVENT_STREAMING_STOPPED, Waiting for inputThread to " - "join..."); - CTX.streaming = false; - CTX.inputThread.join(); - - GL(glDeleteTextures(2, CTX.streamTextures)); - info("ALVR Poll Event: ALVR_EVENT_STREAMING_STOPPED, Stream stopped deleted " - "textures."); + // The saved frame belongs to the decoder that is about to be destroyed. + CTX.lastStreamBuffer = nullptr; + if (g_decoderCreated) { + alvr_destroy_decoder(); + g_decoderCreated = false; } - } - CardboardEyeTextureDescription viewsDescs[2] = {}; - for (auto &viewsDesc : viewsDescs) { - viewsDesc.left_u = 0.0; - viewsDesc.right_u = 1.0; - viewsDesc.top_v = 1.0; - viewsDesc.bottom_v = 0.0; + GL(glDeleteTextures(2, CTX.streamTextures)); + info("ALVR Poll Event: ALVR_EVENT_STREAMING_STOPPED, Stream stopped deleted " + "textures."); } + } + // 4) right after the closing brace of the `while (alvr_poll_event(&event))` loop, + // before `CardboardEyeTextureDescription viewsDescs[2] = {};` + EGLCHK("after event loop"); + + // Update ARCore on the render thread; getPose() then uses its position and correction. + updateARCoreTracking(); + + CardboardEyeTextureDescription viewsDescs[2] = {}; + for (auto &viewsDesc : viewsDescs) { + viewsDesc.left_u = 0.0; + viewsDesc.right_u = 1.0; + viewsDesc.top_v = 1.0; + viewsDesc.bottom_v = 0.0; + } - if (CTX.streaming) { - void *streamHardwareBuffer = nullptr; - - AlvrViewParams dummyViewParams; - auto timestampNs = alvr_get_frame(&dummyViewParams, &streamHardwareBuffer); - - if (timestampNs == -1) { + if (CTX.streaming) { + void *streamHardwareBuffer = nullptr; + uint64_t timestampNs = 0; + + // A new decoded frame is not available on every display refresh: the video usually + // arrives more slowly than the display runs. Returning here without drawing leaves the + // screen showing stale or blank content, which looks like stuttering. Instead keep + // showing the last frame, re-oriented to the current head pose by the reprojection + // below. + bool isNewFrame = alvr_get_frame(×tampNs, &streamHardwareBuffer); + + AlvrVideoFrameMetadata metadata; + if (isNewFrame) { + if (ARCORE_START_AFTER_VIDEO && CTX.videoFramesSeen < ARCORE_START_AFTER_VIDEO_FRAMES && + ++CTX.videoFramesSeen >= ARCORE_START_AFTER_VIDEO_FRAMES) { + startARCoreSession("video is flowing"); + } + CTX.lastStreamBuffer = streamHardwareBuffer; + CTX.lastStreamTimestampNs = timestampNs; + + // v21 no longer returns view params from alvr_get_frame() directly; fetch the + // server-aligned metadata (view params + foveation centers) matched to this frame + // here instead. If there's no match for this exact frame, fall back to the last + // known-good metadata, per the API doc for this function. + if (alvr_report_compositor_start(timestampNs, &metadata)) { + CTX.lastFrameMetadata = metadata; + CTX.hasFrameMetadata = true; + } else if (CTX.hasFrameMetadata) { + metadata = CTX.lastFrameMetadata; + } else { + // No metadata at all yet (first frame) - nothing sensible to reproject against. return; } - - uint32_t swapchainIndices[2] = {0, 0}; - alvr_render_stream_opengl(streamHardwareBuffer, swapchainIndices); - - alvr_report_submit(timestampNs, 0); - - viewsDescs[0].texture = CTX.streamTextures[0]; - viewsDescs[1].texture = CTX.streamTextures[1]; + } else if (CTX.lastStreamBuffer != nullptr && CTX.hasFrameMetadata) { + // No new frame: show the previous one again, with the metadata that belongs to it. + streamHardwareBuffer = CTX.lastStreamBuffer; + timestampNs = CTX.lastStreamTimestampNs; + metadata = CTX.lastFrameMetadata; } else { - AlvrPose pose = getPose(GetBootTimeNano() + VSYNC_QUEUE_INTERVAL_NS); + // Nothing has been decoded yet. + return; + } - AlvrViewInput viewInputs[2] = {}; - for (int eye = 0; eye < 2; eye++) { - float headToEye[3] = {CTX.eyeOffsets[eye], 0.0, 0.0}; - // offset head pos to Eye Position - offsetPosWithQuat(pose.orientation, headToEye, viewInputs[eye].pose.position); + // Current best head-orientation estimate, for reprojecting away the latency between + // when the server rendered this frame and now. The Cardboard SDK's own distortion + // renderer has no reprojection support of its own; this reprojection_rotation is what + // lets v21 do it properly even on this simple client. + AlvrPose currentPose = getPose(GetBootTimeNano() + VSYNC_QUEUE_INTERVAL_NS); + + AlvrStreamViewParams streamViewParams[2] = {}; + for (int eye = 0; eye < 2; eye++) { + streamViewParams[eye].swapchain_index = 0; + streamViewParams[eye].reprojection_rotation = + alvr_rotation_delta(metadata.view_params[eye].pose.orientation, + currentPose.orientation); + streamViewParams[eye].fov = metadata.view_params[eye].fov; + } - viewInputs[eye].pose.orientation = pose.orientation; - viewInputs[eye].fov = getFov((CardboardEye) eye); - viewInputs[eye].swapchain_index = 0; - } - alvr_render_lobby_opengl(viewInputs); + const float(*foveationCenters)[2][2] = + metadata.has_foveation_centers ? &metadata.foveation_center_shifts : nullptr; + + EGLCHK("before alvr_render_stream_opengl()"); + ALVR_GL_CALL( + alvr_render_stream_opengl(streamHardwareBuffer, streamViewParams, foveationCenters)); + EGLCHK("after alvr_render_stream_opengl()"); - viewsDescs[0].texture = CTX.lobbyTextures[0]; - viewsDescs[1].texture = CTX.lobbyTextures[1]; + // The statistics are per decoded frame, so only report frames that are new. + if (isNewFrame) { + alvr_report_submit(timestampNs, 0); } - // Note: the Cardboard SDK does not support reprojection! - // todo: manually implement it? + viewsDescs[0].texture = CTX.streamTextures[0]; + viewsDescs[1].texture = CTX.streamTextures[1]; + } else { + AlvrPose pose = getPose(GetBootTimeNano() + VSYNC_QUEUE_INTERVAL_NS); + + AlvrLobbyViewParams viewInputs[2] = {}; + for (int eye = 0; eye < 2; eye++) { + float headToEye[3] = {CTX.eyeOffsets[eye], 0.0, 0.0}; + // offset head pos to Eye Position + offsetPosWithQuat( + pose.orientation, headToEye, viewInputs[eye].view_params.pose.position); + + viewInputs[eye].view_params.pose.orientation = pose.orientation; + viewInputs[eye].view_params.fov = getFov((CardboardEye) eye); + viewInputs[eye].swapchain_index = 0; + } + static int lobbyTexLogs = 0; + if (lobbyTexLogs++ < 3) { + LOG("LOBBYTEX before lobby render: names=%u,%u isTexture=%d,%d", + (unsigned) CTX.lobbyTextures[0], + (unsigned) CTX.lobbyTextures[1], + (int) glIsTexture(CTX.lobbyTextures[0]), + (int) glIsTexture(CTX.lobbyTextures[1])); + } + EGLCHK("before alvr_render_lobby_opengl()"); + ALVR_GL_CALL(alvr_render_lobby_opengl(viewInputs, /*render_background=*/true)); + EGLCHK("after alvr_render_lobby_opengl()"); - // info("nativeRendered: Rendering to Display..."); - CardboardDistortionRenderer_renderEyeToDisplay(CTX.distortionRenderer, - 0, - 0, - 0, - CTX.screenWidth, - CTX.screenHeight, - &viewsDescs[0], - &viewsDescs[1]); - } catch (const json::exception &e) { - error(std::string(std::string(__FUNCTION__) + std::string(__FILE_NAME__) + - std::string(e.what())) - .c_str()); + viewsDescs[0].texture = CTX.lobbyTextures[0]; + viewsDescs[1].texture = CTX.lobbyTextures[1]; } + EGLCHK("before CardboardDistortionRenderer_renderEyeToDisplay()"); + CardboardDistortionRenderer_renderEyeToDisplay(CTX.distortionRenderer, + 0, + 0, + 0, + CTX.screenWidth, + CTX.screenHeight, + &viewsDescs[0], + &viewsDescs[1]); + EGLCHK("after CardboardDistortionRenderer_renderEyeToDisplay()"); } extern "C" JNIEXPORT void JNICALL diff --git a/code/mobile/android/PhoneVR/app/src/main/cpp/head_orientation_filter.h b/code/mobile/android/PhoneVR/app/src/main/cpp/head_orientation_filter.h new file mode 100644 index 00000000..e750b02e --- /dev/null +++ b/code/mobile/android/PhoneVR/app/src/main/cpp/head_orientation_filter.h @@ -0,0 +1,322 @@ +#ifndef PHONEVR_HEAD_ORIENTATION_FILTER_H +#define PHONEVR_HEAD_ORIENTATION_FILTER_H + +// 3DoF head orientation: Mahony complementary filter plus a stillness-based gyro bias estimator. +// +// FRAMES (same as OpenXR / ALVR, which is what the rest of alvr_main.cpp works in) +// Head frame: x = right, y = up, z = towards the wearer's face, forward = -z. +// World frame: y = up (gravity), x/z horizontal. Identity = looking along -z. +// The orientation returned is world_from_head, as a quaternion (x, y, z, w). +// +// INPUT: gyro (rad/s) and accelerometer (m/s^2, gravity included), already in the head frame, and +// the gyro timestamp in ns. Why the bias estimator exists: a gyro reads a small constant offset +// when still; integrating it makes the yaw turn slowly forever. Gravity corrects pitch and roll, +// nothing corrects yaw, so the offset has to be measured: whenever the head has been still for +// about half a second, the mean gyro reading in that window IS the offset. +// +// Thread-safe: sensor thread calls onGyroscope/onAccelerometer, any thread reads orientationAt(). + +#include +#include +#include +#include +#include + +namespace phonevr { + +struct Quatd { + double x = 0, y = 0, z = 0, w = 1; +}; + +class HeadOrientationFilter { +public: + struct Config { + // Mahony (accelerometer -> pitch/roll) + double kp = 1.0; // proportional gain, 1/s + double ki = 0.0; // integral gain; 0 = off (the bias estimator handles drift) + double accelTolerance = 3.0; // m/s^2 away from g at which the accelerometer weight is 0 + + // Stillness-based gyro bias + int biasWindowSamples = 100; // ~0.5 s at 200 Hz + double stillGyroStdDps = 0.6; // max gyro std-dev inside the window to count as still + double stillAccelStd = 0.12; // max std-dev of |accel| (m/s^2) + double stillAccelMeanTol = 0.4;// max | mean|accel| - g | (m/s^2) + double bootstrapMeanDps = 5.0; // before the first estimate: max |mean gyro| + double meanDevDps = 1.5; // afterwards: max distance of the window mean from the bias + double biasGain = 0.1; // blend per accepted (non-overlapping) window + + // Prediction: orientationAt() extrapolates with the latest angular velocity, at most this far + double maxPredictionSec = 0.1; + }; + + struct Diagnostics { + double biasDps[3]; + bool biasReady; + }; + + // (Two constructors rather than "cfg = Config()": a default argument can't use Config's + // member initializers before the enclosing class is complete.) + HeadOrientationFilter() : HeadOrientationFilter(Config()) {} + explicit HeadOrientationFilter(const Config &cfg) : cfg_(cfg), bias_(cfg) {} + + void onAccelerometer(double x, double y, double z) { + std::lock_guard lock(m_); + ax_ = x; ay_ = y; az_ = z; + hasAccel_ = true; + } + + // Main step: call for every gyro sample. + void onGyroscope(double gx, double gy, double gz, int64_t timestampNs) { + std::lock_guard lock(m_); + if (!hasAccel_) return; + if (!initialised_) { + initFromAccel(); + lastTs_ = timestampNs; + initialised_ = true; + recenterLocked(); + return; + } + double dt = (double) (timestampNs - lastTs_) * 1e-9; + lastTs_ = timestampNs; + if (dt <= 0.0 || dt > 0.1) return; // clock hiccup or pause: don't integrate + + double aMag = std::sqrt(ax_ * ax_ + ay_ * ay_ + az_ * az_); + bias_.push(gx, gy, gz, aMag); + + double wx = gx - bias_.b[0]; + double wy = gy - bias_.b[1]; + double wz = gz - bias_.b[2]; + rateX_ = wx; rateY_ = wy; rateZ_ = wz; // for prediction (bias-corrected, no accel term) + + // "up" in the head frame = second row of the rotation matrix + double vx = 2.0 * (qx_ * qy_ + qw_ * qz_); + double vy = 1.0 - 2.0 * (qx_ * qx_ + qz_ * qz_); + double vz = 2.0 * (qy_ * qz_ - qw_ * qx_); + + // accelerometer error: a_unit x v, weighted down while the head is accelerating + if (aMag > 1e-3) { + double weight = std::max(0.0, 1.0 - std::fabs(aMag - kG) / cfg_.accelTolerance); + if (weight > 0.0) { + double nx = ax_ / aMag, ny = ay_ / aMag, nz = az_ / aMag; + double ex = (ny * vz - nz * vy) * weight; + double ey = (nz * vx - nx * vz) * weight; + double ez = (nx * vy - ny * vx) * weight; + if (cfg_.ki > 0.0) { + eIntX_ += ex * cfg_.ki * dt; eIntY_ += ey * cfg_.ki * dt; eIntZ_ += ez * cfg_.ki * dt; + wx += eIntX_; wy += eIntY_; wz += eIntZ_; + } + wx += cfg_.kp * ex; wy += cfg_.kp * ey; wz += cfg_.kp * ez; + } + } + + // integrate q' = 0.5 * q (x) (0, w) + double h = 0.5 * dt; + double nw = qw_ + h * (-qx_ * wx - qy_ * wy - qz_ * wz); + double nx = qx_ + h * (qw_ * wx + qy_ * wz - qz_ * wy); + double ny = qy_ + h * (qw_ * wy - qx_ * wz + qz_ * wx); + double nz = qz_ + h * (qw_ * wz + qx_ * wy - qy_ * wx); + double inv = 1.0 / std::sqrt(nw * nw + nx * nx + ny * ny + nz * nz); + qw_ = nw * inv; qx_ = nx * inv; qy_ = ny * inv; qz_ = nz * inv; + } + + bool ready() const { + std::lock_guard lock(m_); + return initialised_; + } + + // Orientation at timestampNs (same clock as the gyro timestamps; CLOCK_BOOTTIME on Android), + // extrapolated from the last gyro sample with the current angular velocity. + Quatd orientationAt(int64_t timestampNs) const { + std::lock_guard lock(m_); + double dt = std::clamp((double) (timestampNs - lastTs_) * 1e-9, 0.0, cfg_.maxPredictionSec); + double w[4] = {qw_, qx_, qy_, qz_}; + double speed = std::sqrt(rateX_ * rateX_ + rateY_ * rateY_ + rateZ_ * rateZ_); + if (dt > 0.0 && speed > 1e-6) { + double half = 0.5 * speed * dt, s = std::sin(half) / speed, c = std::cos(half); + double dw = c, dx = rateX_ * s, dy = rateY_ * s, dz = rateZ_ * s; + double r[4]; + mul(w, dw, dx, dy, dz, r); + for (int i = 0; i < 4; i++) w[i] = r[i]; + } + double out[4]; + mul2(ow_, 0.0, oy_, 0.0, w, out); + return {out[1], out[2], out[3], out[0]}; + } + + // Makes the direction you are looking now "straight ahead". Output only; filter untouched. + void recenter() { + std::lock_guard lock(m_); + recenterLocked(); + } + + // Start over (e.g. on resume). Keeps the gyro bias estimate. + void reset() { + std::lock_guard lock(m_); + initialised_ = false; + hasAccel_ = false; + qw_ = 1; qx_ = qy_ = qz_ = 0; + ow_ = 1; oy_ = 0; + eIntX_ = eIntY_ = eIntZ_ = 0; + rateX_ = rateY_ = rateZ_ = 0; + } + + // rad/s. Can be saved per device and restored to skip the warm-up. + void getGyroBias(double out[3]) const { + std::lock_guard lock(m_); + for (int i = 0; i < 3; i++) out[i] = bias_.b[i]; + } + void setGyroBias(double x, double y, double z) { + std::lock_guard lock(m_); + bias_.b[0] = x; bias_.b[1] = y; bias_.b[2] = z; + bias_.ready = true; + } + + Diagnostics diagnostics() const { + std::lock_guard lock(m_); + Diagnostics d; + for (int i = 0; i < 3; i++) d.biasDps[i] = bias_.b[i] * 180.0 / M_PI; + d.biasReady = bias_.ready; + return d; + } + +private: + static constexpr double kG = 9.80665; + + // r = a (x) (w, x, y, z) with a = {w,x,y,z} + static void mul(const double a[4], double bw, double bx, double by, double bz, double r[4]) { + r[0] = a[0] * bw - a[1] * bx - a[2] * by - a[3] * bz; + r[1] = a[0] * bx + a[1] * bw + a[2] * bz - a[3] * by; + r[2] = a[0] * by - a[1] * bz + a[2] * bw + a[3] * bx; + r[3] = a[0] * bz + a[1] * by - a[2] * bx + a[3] * bw; + } + // r = (aw, ax, ay, az) (x) b with b = {w,x,y,z} + static void mul2(double aw, double ax, double ay, double az, const double b[4], double r[4]) { + r[0] = aw * b[0] - ax * b[1] - ay * b[2] - az * b[3]; + r[1] = aw * b[1] + ax * b[0] + ay * b[3] - az * b[2]; + r[2] = aw * b[2] - ax * b[3] + ay * b[0] + az * b[1]; + r[3] = aw * b[3] + ax * b[2] - ay * b[1] + az * b[0]; + } + + void initFromAccel() { + double n = std::sqrt(ax_ * ax_ + ay_ * ay_ + az_ * az_); + double ux = ax_ / n, uy = ay_ / n, uz = az_ / n; + // shortest rotation taking the measured "up" (head frame) onto world +y + if (1.0 + uy < 1e-6) { + qw_ = 0; qx_ = 1; qy_ = 0; qz_ = 0; + } else { + double w = 1.0 + uy, x = -uz, z = ux; + double m = 1.0 / std::sqrt(w * w + x * x + z * z); + qw_ = w * m; qx_ = x * m; qy_ = 0; qz_ = z * m; + } + } + + void recenterLocked() { + // forward = -(third column of R) + double fx = -2.0 * (qx_ * qz_ + qw_ * qy_); + double fz = -(1.0 - 2.0 * (qx_ * qx_ + qy_ * qy_)); + if (std::hypot(fx, fz) < 0.2) return; // looking straight up/down: yaw undefined + double yaw = std::atan2(-fx, -fz); // rotation about +y + ow_ = std::cos(yaw / 2.0); + oy_ = -std::sin(yaw / 2.0); // Ry(-yaw) + } + + // Estimates the gyro bias whenever the head is still. Works on raw gyro (before bias + // subtraction). O(1) per sample: ring buffer with running sums. + struct BiasEstimator { + explicit BiasEstimator(const Config &c) + : cfg(c), n(std::max(8, c.biasWindowSamples)), stride(std::max(1, n / 4)) { + for (auto &v : buf) v.assign(n, 0.0); + am.assign(n, 0.0); + } + + double b[3] = {0, 0, 0}; // bias, rad/s + bool ready = false; + + void push(double gx, double gy, double gz, double aMag) { + double g[3] = {gx, gy, gz}; + if (count == n) { + for (int i = 0; i < 3; i++) { double o = buf[i][idx]; sum[i] -= o; sumSq[i] -= o * o; } + double o = am[idx]; aSum -= o; aSumSq -= o * o; + } else { + count++; + } + for (int i = 0; i < 3; i++) { buf[i][idx] = g[i]; sum[i] += g[i]; sumSq[i] += g[i] * g[i]; } + am[idx] = aMag; aSum += aMag; aSumSq += aMag * aMag; + if (++idx == n) { idx = 0; recompute(); } // stop float drift in the running sums + sinceEval++; sinceUsed++; + if (count < n || sinceEval < stride) return; + sinceEval = 0; + evaluate(); + } + + private: + Config cfg; + int n, stride; + std::vector buf[3]; + std::vector am; + double sum[3] = {0, 0, 0}, sumSq[3] = {0, 0, 0}; + double aSum = 0, aSumSq = 0; + int idx = 0, count = 0, sinceEval = 0, sinceUsed = 0, streak = 0; + static constexpr int kRebaseStreak = 30; + + static double rad(double deg) { return deg * M_PI / 180.0; } + + void recompute() { + for (int i = 0; i < 3; i++) { + double s = 0, q = 0; + for (double v : buf[i]) { s += v; q += v * v; } + sum[i] = s; sumSq[i] = q; + } + double s = 0, q = 0; + for (double v : am) { s += v; q += v * v; } + aSum = s; aSumSq = q; + } + + void evaluate() { + double inv = 1.0 / n; + double aMean = aSum * inv; + double aVar = aSumSq * inv - aMean * aMean; + if (std::fabs(aMean - kG) > cfg.stillAccelMeanTol || aVar > cfg.stillAccelStd * cfg.stillAccelStd) { + streak = 0; + return; + } + double stdTol = rad(cfg.stillGyroStdDps); + double tol = rad(ready ? cfg.meanDevDps : cfg.bootstrapMeanDps); + bool meanOk = true; + for (int i = 0; i < 3; i++) { + double mean = sum[i] * inv; + if (sumSq[i] * inv - mean * mean > stdTol * stdTol) { streak = 0; return; } + if (std::fabs(mean - (ready ? b[i] : 0.0)) > tol) meanOk = false; + } + bool force = false; + if (!meanOk) { + // Still and steady but the mean moved: after ~5 s of that, believe it. + if (!ready || ++streak < kRebaseStreak) return; + force = true; + } + if (ready && !force && sinceUsed < n) return; // don't reuse the same samples + double alpha = (!ready || force) ? 1.0 : cfg.biasGain; + for (int i = 0; i < 3; i++) b[i] += alpha * (sum[i] * inv - b[i]); + ready = true; + sinceUsed = 0; + streak = 0; + } + }; + + Config cfg_; + BiasEstimator bias_; + mutable std::mutex m_; + + double qw_ = 1, qx_ = 0, qy_ = 0, qz_ = 0; + double ow_ = 1, oy_ = 0; // recenter offset (rotation about world up), output only + bool initialised_ = false; + int64_t lastTs_ = 0; + bool hasAccel_ = false; + double ax_ = 0, ay_ = 0, az_ = 0; + double eIntX_ = 0, eIntY_ = 0, eIntZ_ = 0; + double rateX_ = 0, rateY_ = 0, rateZ_ = 0; +}; + +} // namespace phonevr + +#endif // PHONEVR_HEAD_ORIENTATION_FILTER_H diff --git a/code/mobile/android/PhoneVR/app/src/main/cpp/head_sensor_source.h b/code/mobile/android/PhoneVR/app/src/main/cpp/head_sensor_source.h new file mode 100644 index 00000000..f788bce5 --- /dev/null +++ b/code/mobile/android/PhoneVR/app/src/main/cpp/head_sensor_source.h @@ -0,0 +1,154 @@ +#ifndef PHONEVR_HEAD_SENSOR_SOURCE_H +#define PHONEVR_HEAD_SENSOR_SOURCE_H + +// Reads the phone's gyroscope and accelerometer through the NDK sensor API on its own thread and +// feeds a HeadOrientationFilter. start() when the app resumes, stop() when it pauses. +// +// AXES: the filter wants the head frame (x right, y up, z towards the wearer's face). The phone +// sits in the headset in landscape, with gravity along its x axis. Which way round it sits +// (the two landscape directions) is read from the accelerometer, so nothing here depends on +// what Cardboard calls "landscape left": +// device +x points up -> head = (-y, x, z) +// device +x points down -> head = ( y, -x, z) +// If the phone is later turned 180 degrees in the headset, the mapping flips (after the new +// direction has been steady for about a second) and the filter restarts, keeping the gyro bias. +// +// SENSORS: TYPE_GYROSCOPE_UNCALIBRATED (raw gyro; the OS's own bias guess is ignored because the +// filter estimates the bias itself), TYPE_ACCELEROMETER. Both at 200 Hz (5000 us), the most +// Android 12+ allows without the HIGH_SAMPLING_RATE_SENSORS permission. + +#include +#include +#include + +#include +#include +#include + +#include "head_orientation_filter.h" + +namespace phonevr { + +class HeadSensorSource { +public: + ~HeadSensorSource() { stop(); } + + void start(HeadOrientationFilter *filter, int rateUs = 5000) { + if (running_.exchange(true)) return; + filter_ = filter; + rateUs_ = rateUs; + thread_ = std::thread([this] { run(); }); + } + + void stop() { + if (!running_.exchange(false)) return; + if (thread_.joinable()) thread_.join(); + } + +private: + static constexpr int kLooperId = 3; + static constexpr int kTypeGyroUncalibrated = 16; // ASENSOR_TYPE_GYROSCOPE_UNCALIBRATED (API 26+) + static constexpr int kFlipSamples = 200; // ~1 s at 200 Hz + + std::thread thread_; + std::atomic running_{false}; + HeadOrientationFilter *filter_ = nullptr; + int rateUs_ = 5000; + + // Mapping state (sensor thread only) + bool mapKnown_ = false; + int sign_ = 1; // +1: device +x points up; -1: points down + int flipCount_ = 0; // consecutive accelerometer samples that disagree with sign_ + + static void logi(const char *msg, double a = 0, double b = 0) { + __android_log_print(ANDROID_LOG_INFO, "PhoneVR_HeadSensors", msg, a, b); + } + + void remap(double x, double y, double z, double out[3]) const { + if (sign_ > 0) { out[0] = -y; out[1] = x; out[2] = z; } + else { out[0] = y; out[1] = -x; out[2] = z; } + } + + void handleAccelerometer(const ASensorEvent &e) { + double x = e.data[0], y = e.data[1], z = e.data[2]; + bool upright = std::fabs(x) > 5.0 && std::fabs(x) > std::fabs(z); // headset-like pose + if (!mapKnown_) { + if (!upright) return; + sign_ = x > 0 ? 1 : -1; + mapKnown_ = true; + logi("phone in headset, device +x points %.0f (1 = up, -1 = down)", sign_); + } else if (upright && std::fabs(x) > 7.0 && (x > 0 ? 1 : -1) != sign_) { + if (++flipCount_ >= kFlipSamples) { + // Turned 180 degrees in the headset. A fixed device-frame gyro bias changes sign + // in head x and y, so carry it across. + double b[3]; + filter_->getGyroBias(b); + sign_ = -sign_; + flipCount_ = 0; + filter_->reset(); + filter_->setGyroBias(-b[0], -b[1], b[2]); + logi("phone turned round, device +x now points %.0f", sign_); + } + } else { + flipCount_ = 0; + } + double h[3]; + remap(x, y, z, h); + filter_->onAccelerometer(h[0], h[1], h[2]); + } + + void handleGyroscope(const ASensorEvent &e) { + if (!mapKnown_) return; + double h[3]; + remap(e.data[0], e.data[1], e.data[2], h); + filter_->onGyroscope(h[0], h[1], h[2], e.timestamp); // CLOCK_BOOTTIME, like GetBootTimeNano() + } + + void run() { + ALooper *looper = ALooper_prepare(ALOOPER_PREPARE_ALLOW_NON_CALLBACKS); + ASensorManager *mgr = ASensorManager_getInstance(); // minSdk 24: the package-name variant needs 26 + int gyroType = kTypeGyroUncalibrated; + const ASensor *gyro = ASensorManager_getDefaultSensor(mgr, gyroType); + if (!gyro) { + gyroType = ASENSOR_TYPE_GYROSCOPE; + gyro = ASensorManager_getDefaultSensor(mgr, gyroType); + } + const ASensor *accel = ASensorManager_getDefaultSensor(mgr, ASENSOR_TYPE_ACCELEROMETER); + if (!gyro || !accel) { + logi("no gyroscope or accelerometer; own head filter disabled"); + return; + } + ASensorEventQueue *queue = + ASensorManager_createEventQueue(mgr, looper, kLooperId, nullptr, nullptr); + if (!queue) { + logi("could not create sensor event queue"); + return; + } + ASensorEventQueue_enableSensor(queue, gyro); + ASensorEventQueue_setEventRate(queue, gyro, rateUs_); + ASensorEventQueue_enableSensor(queue, accel); + ASensorEventQueue_setEventRate(queue, accel, rateUs_); + + mapKnown_ = false; + flipCount_ = 0; + while (running_) { + int ident = ALooper_pollAll(100, nullptr, nullptr, nullptr); + if (ident != kLooperId) continue; + ASensorEvent ev[32]; + ssize_t n; + while ((n = ASensorEventQueue_getEvents(queue, ev, 32)) > 0) { + for (ssize_t i = 0; i < n; i++) { + if (ev[i].type == ASENSOR_TYPE_ACCELEROMETER) handleAccelerometer(ev[i]); + else if (ev[i].type == gyroType) handleGyroscope(ev[i]); + } + } + } + ASensorEventQueue_disableSensor(queue, gyro); + ASensorEventQueue_disableSensor(queue, accel); + ASensorManager_destroyEventQueue(mgr, queue); + } +}; + +} // namespace phonevr + +#endif // PHONEVR_HEAD_SENSOR_SOURCE_H diff --git a/code/mobile/android/PhoneVR/app/src/main/cpp/neck_model.h b/code/mobile/android/PhoneVR/app/src/main/cpp/neck_model.h new file mode 100644 index 00000000..722e11fa --- /dev/null +++ b/code/mobile/android/PhoneVR/app/src/main/cpp/neck_model.h @@ -0,0 +1,49 @@ +#ifndef PHONEVR_NECK_MODEL_H +#define PHONEVR_NECK_MODEL_H + +// Neck model for seated 3DoF: fakes the small eye movement you get when the head pivots on the +// neck, so looking down at a dashboard or tilting towards a mirror gives parallax even though +// there is no positional tracking. It is not tracking: leaning the body or ducking does nothing. +// +// The eyes sit at e = (0, up, -forward) from the neck pivot, in the head frame (x right, y up, +// forward = -z). With head orientation R (world_from_head) the eyes are at R*e. The returned +// offset is that minus where the eyes would be with the same heading and no pitch or roll: +// offset = R*e - Ry(yaw)*e +// so it is exactly zero when you look straight ahead, and ignores turning the head left and right +// (yaw has no fixed meaning in this app: it starts wherever the app launched and can be nudged +// by the ARCore yaw correction, so including it would make the view jump when that happens). +// +// Frame: y up, x/z horizontal (same as ALVR / OpenXR). Result in metres. + +#include + +namespace phonevr { + +// q = world_from_head orientation (x, y, z, w). lastYawRad is state kept by the caller: the +// heading is undefined when looking straight up or down, so it is held until the head levels out. +inline void neckModelOffset(float qx, float qy, float qz, float qw, float eyeUpM, + float eyeForwardM, float &lastYawRad, float out[3]) { + // forward = -(third column of R); heading is its direction on the horizontal plane + float fx = -2.f * (qx * qz + qw * qy); + float fz = -(1.f - 2.f * (qx * qx + qy * qy)); + if (std::hypot(fx, fz) >= 0.25f) { + lastYawRad = std::atan2(-fx, -fz); + } + float s = std::sin(lastYawRad), c = std::cos(lastYawRad); + + // R*e = up * (second column of R) - forward * (third column of R) + float re[3]; + re[0] = eyeUpM * 2.f * (qx * qy - qw * qz) - eyeForwardM * 2.f * (qx * qz + qw * qy); + re[1] = eyeUpM * (1.f - 2.f * (qx * qx + qz * qz)) - eyeForwardM * 2.f * (qy * qz - qw * qx); + re[2] = eyeUpM * 2.f * (qy * qz + qw * qx) - eyeForwardM * (1.f - 2.f * (qx * qx + qy * qy)); + + // Ry(yaw)*e for e = (0, up, -forward) + float neutral[3] = {-eyeForwardM * s, eyeUpM, -eyeForwardM * c}; + for (int i = 0; i < 3; i++) { + out[i] = re[i] - neutral[i]; + } +} + +} // namespace phonevr + +#endif // PHONEVR_NECK_MODEL_H diff --git a/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/ALVRActivity.java b/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/ALVRActivity.java index b5b09f56..365f93c8 100644 --- a/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/ALVRActivity.java +++ b/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/ALVRActivity.java @@ -105,7 +105,21 @@ public void onCreate(Bundle savedInstance) { float refreshRate = display.getRefreshRate(); Log.i(TAG, "Refresh rate: " + refreshRate); - initializeNative(width, height, refreshRate); + // Options chosen on the Settings screen (InitActivity -> Settings). + SharedPreferences appSettings = getSharedPreferences(SettingsActivity.PREFS, MODE_PRIVATE); + boolean enableARCore = + appSettings.getBoolean( + SettingsActivity.KEY_ARCORE_ENABLED, + SettingsActivity.DEFAULT_ARCORE_ENABLED); + Log.i(TAG, "ARCore enabled in settings: " + enableARCore); + + initializeNative(width, height, refreshRate, enableARCore); + setStreamFovScaleNative( + appSettings.getFloat( + SettingsActivity.KEY_FOV_SCALE, SettingsActivity.DEFAULT_FOV_SCALE)); + setFloorModeNative( + appSettings.getInt( + SettingsActivity.KEY_FLOOR_MODE, SettingsActivity.DEFAULT_FLOOR_MODE)); setContentView(R.layout.activity_vr); glView = findViewById(R.id.surface_view); @@ -285,7 +299,11 @@ private void setImmersiveSticky() { } private native void initializeNative( - int screenWidth, int screenHeight, float screenRefreshRate); + int screenWidth, int screenHeight, float screenRefreshRate, boolean enableARCore); + + private native void setStreamFovScaleNative(float scale); + + private native void setFloorModeNative(int mode); private native void destroyNative(); diff --git a/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/InitActivity.kt b/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/InitActivity.kt index 1e951ca7..442d1b41 100644 --- a/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/InitActivity.kt +++ b/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/InitActivity.kt @@ -58,6 +58,7 @@ class InitActivity : AppCompatActivity() { else ALVRActivity::class.java startActivity(Intent(this, intentClass)) + if (intentClass == ALVRActivity::class.java) finish() } swVRemember.setOnCheckedChangeListener { _, isChecked -> @@ -68,6 +69,10 @@ class InitActivity : AppCompatActivity() { } } + fun btOnClickSettings(view: View) { + startActivity(Intent(this, SettingsActivity::class.java)) + } + fun btOnClickALVRStreamer(view: View) { val prefs = getSharedPreferences("prefs", MODE_PRIVATE) // Save choice after first open. alvr_server as boolean @@ -76,6 +81,7 @@ class InitActivity : AppCompatActivity() { val intent = Intent(this, ALVRActivity::class.java) startActivity(intent) + finish() } fun btOnClickPVRStreamer(view: View) { diff --git a/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/SettingsActivity.java b/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/SettingsActivity.java new file mode 100644 index 00000000..ee0062a2 --- /dev/null +++ b/code/mobile/android/PhoneVR/app/src/main/java/viritualisres/phonevr/SettingsActivity.java @@ -0,0 +1,103 @@ +/* (C)2026 */ +package viritualisres.phonevr; + +import android.content.SharedPreferences; +import android.os.Bundle; +import android.view.View; +import android.widget.AdapterView; +import android.widget.ArrayAdapter; +import android.widget.SeekBar; +import android.widget.Spinner; +import android.widget.TextView; +import androidx.appcompat.app.AppCompatActivity; +import androidx.appcompat.widget.SwitchCompat; + +/** + * The settings screen shown from the start screen, before VR starts. Every change is saved at once + * to the "settings" preferences and read by ALVRActivity when VR starts. + */ +public class SettingsActivity extends AppCompatActivity { + + // The same file ALVRActivity already uses for max_brightness. + static final String PREFS = "settings"; + + static final String KEY_ARCORE_ENABLED = "arcore_enabled"; + static final String KEY_FOV_SCALE = "fov_scale"; + static final String KEY_FLOOR_MODE = "floor_mode"; + + static final boolean DEFAULT_ARCORE_ENABLED = true; + static final float DEFAULT_FOV_SCALE = 0.8f; + static final int DEFAULT_FLOOR_MODE = 2; + + // The slider covers 60% .. 100% in 1% steps (progress 0 .. 40). + private static final int FOV_MIN_PERCENT = 60; + private static final int FOV_MAX_PERCENT = 100; + + private SharedPreferences prefs; + private TextView fovValue; + + @Override + protected void onCreate(Bundle savedInstanceState) { + super.onCreate(savedInstanceState); + setContentView(R.layout.activity_settings); + prefs = getSharedPreferences(PREFS, MODE_PRIVATE); + + // ARCore on/off + SwitchCompat arcoreSwitch = findViewById(R.id.settings_arcore_switch); + arcoreSwitch.setChecked(prefs.getBoolean(KEY_ARCORE_ENABLED, DEFAULT_ARCORE_ENABLED)); + arcoreSwitch.setOnCheckedChangeListener( + (button, isChecked) -> + prefs.edit().putBoolean(KEY_ARCORE_ENABLED, isChecked).apply()); + + // Field of view slider + fovValue = findViewById(R.id.settings_fov_value); + SeekBar fovSeek = findViewById(R.id.settings_fov_seekbar); + fovSeek.setMax(FOV_MAX_PERCENT - FOV_MIN_PERCENT); + int percent = Math.round(prefs.getFloat(KEY_FOV_SCALE, DEFAULT_FOV_SCALE) * 100f); + percent = Math.max(FOV_MIN_PERCENT, Math.min(FOV_MAX_PERCENT, percent)); + fovSeek.setProgress(percent - FOV_MIN_PERCENT); + showFovPercent(percent); + fovSeek.setOnSeekBarChangeListener( + new SeekBar.OnSeekBarChangeListener() { + @Override + public void onProgressChanged(SeekBar bar, int progress, boolean fromUser) { + int p = FOV_MIN_PERCENT + progress; + showFovPercent(p); + if (fromUser) { + prefs.edit().putFloat(KEY_FOV_SCALE, p / 100f).apply(); + } + } + + @Override + public void onStartTrackingTouch(SeekBar bar) {} + + @Override + public void onStopTrackingTouch(SeekBar bar) {} + }); + + // Floor (height) mode + Spinner floorSpinner = findViewById(R.id.settings_floor_spinner); + ArrayAdapter adapter = + ArrayAdapter.createFromResource( + this, R.array.floor_mode_choices, android.R.layout.simple_spinner_item); + adapter.setDropDownViewResource(android.R.layout.simple_spinner_dropdown_item); + floorSpinner.setAdapter(adapter); + int mode = prefs.getInt(KEY_FLOOR_MODE, DEFAULT_FLOOR_MODE); + floorSpinner.setSelection(Math.max(0, Math.min(adapter.getCount() - 1, mode))); + floorSpinner.setOnItemSelectedListener( + new AdapterView.OnItemSelectedListener() { + @Override + public void onItemSelected( + AdapterView parent, View view, int position, long id) { + prefs.edit().putInt(KEY_FLOOR_MODE, position).apply(); + } + + @Override + public void onNothingSelected(AdapterView parent) {} + }); + } + + private void showFovPercent(int percent) { + fovValue.setText(percent + "%"); + } +} diff --git a/code/mobile/android/PhoneVR/app/src/main/res/layout/activity_init.xml b/code/mobile/android/PhoneVR/app/src/main/res/layout/activity_init.xml index a3da1607..65ca0509 100644 --- a/code/mobile/android/PhoneVR/app/src/main/res/layout/activity_init.xml +++ b/code/mobile/android/PhoneVR/app/src/main/res/layout/activity_init.xml @@ -7,6 +7,17 @@ tools:context=".InitActivity"> +