Merge pull request #6949 from RaananW/getScene-when-no-grip

WebXR changes

dist/preview release/what's new.md

@@ -1,6 +1,7 @@
 # 4.1.0
 
 ## Major updates
+
 - WIP: Node Material (NEED DOC OR SAMPLES) ([Deltakosh](https://github.com/deltakosh/))
 - WIP: Node material editor (NEED OR AND VIDEOS) ([Deltakosh](https://github.com/deltakosh/) / [TrevorDev](https://github.com/TrevorDev))
 - WIP: WebGPU support (NEED DOC OR SAMPLES) ([Sebavan](https://github.com/sebavan/))
@@ -17,6 +18,7 @@
 ## Updates
 
 ### General
+
 - Added support for dual shock gamepads ([Deltakosh](https://github.com/deltakosh/))
 - Support Vive Focus 3Dof controller ([TrevorDev](https://github.com/TrevorDev))
 - Planar positioning support for GizmoManager ([Balupg](https://github.com/balupg))
@@ -38,6 +40,7 @@
 - Added optional parameter to use Euler angles in planeRotationGizmo ([CedricGuillemet](https://github.com/CedricGuillemet))
 
 ### Engine
+
 - Improved instanceMesh with user defined custom buffers [Doc](https://doc.babylonjs.com/how_to/how_to_use_instances#custom-buffers) ([Deltakosh](https://github.com/deltakosh/))
 - Morph targets now can morph UV channel as well ([Deltakosh](https://github.com/deltakosh/))
 - Added MorphTarget support to the DepthRenderer, GeometryBufferRenderer and OutlineRenderer ([MarkusBillharz](https://github.com/MarkusBillharz))
@@ -46,6 +49,7 @@
 - Added support for `vertexSource` and `fragmentSource` parameters to `ShaderMaterial` ([Deltakosh](https://github.com/deltakosh/))
 
 ### Inspector
+
 - Added support for Euler edition only for angles (can be turned off in the new inspector settings) ([Deltakosh](https://github.com/deltakosh/))
 - Added an option to ignore backfaces for picking (can be turned on and off in the new inspector settings) ([Deltakosh](https://github.com/deltakosh/))
 - Added support for `ShadowGenerator` ([Deltakosh](https://github.com/deltakosh/))
@@ -60,6 +64,7 @@
 - Added support for CreateScreenshotUsingRenderTarget ([13djwright](https://github.com/13djwright/))
 
 ### Tools
+
 - Added `Tools.CreateScreenshotAsync` and `Tools.CreateScreenshotUsingRenderTargetAsync` ([mehmetoguzderin](https://github.com/mehmetoguzderin/))
 - Added `Color3.toHSV()`, `Color3.toHSVToRef()` and `Color3.HSVtoRGBToRef()` ([Deltakosh](https://github.com/deltakosh/))
 - Added `ShadowGenerator.onAfterShadowMapRenderObservable` and `ShadowGenerator.onAfterShadowMapMeshRenderObservable` ([Deltakosh](https://github.com/deltakosh/))
@@ -68,18 +73,21 @@
 - Added RGBD Texture tools [Sebavan](https://github.com/sebavan/)
 
 ### Meshes
+
 - Added `TransformNode.instantiateHierarchy()` which try to instantiate (or clone) a node and its entire hiearchy ([Deltakosh](https://github.com/deltakosh/))
 - Added new CreateTiledPlane and CreateTiledBox ([JohnK](https://github.com/BabylonJSGuide/))
 - Added absolute scaling and rotation getters ([haroldma](https://github.com/haroldma))
 - Added `BILLBOARDMODE_USE_POSITION` flag to billboards allowing use of camera positioning instead of orientation for mesh rotation ([delaneyj](https://github.com/delaneyj))
 
 ### Physics
+
 - Update Ammo.js library to support global collision contact callbacks ([MackeyK24](https://github.com/MackeyK24/))
 - Update Ammo.js library to allow native capsule shape impostors ([MackeyK24](https://github.com/MackeyK24/))
 - Update Ammo.js library to allow your own broadphase overlapping pair cache ([MackeyK24](https://github.com/MackeyK24/))
 - Update Ammo.js library and AmmoJS plugin to support ellipsoid ([CedricGuillemet](https://github.com/CedricGuillemet/))
 
 ### Loaders
+
 - Added support for non-float accessors in animation data for glTF loader. ([bghgary](https://github.com/bghgary))
 - Support loading cube data in the .basis loader ([TrevorDev](https://github.com/TrevorDev))
 - Load glTF extras into BJS metadata ([pjoe](https://github.com/pjoe))
@@ -88,6 +96,7 @@
 - Added a flag to enable/disable creation of instances for glTF loader. ([bghgary](https://github.com/bghgary))
 
 ### Materials
+
 - Added `ShaderMaterial.setColor4Array` ([JonathanTron](https://github.com/JonathanTron/))
 - Added `ShaderMaterial.setArray4` ([JonathanTron](https://github.com/JonathanTron/))
 - Added `scene.environmentIntensity` to control the IBL strength overall in a scene ([Sebavan](https://github.com/sebavan/))
@@ -95,21 +104,32 @@
 - Added `pbrBRDFConfiguration.useSpecularGlossinessInputEnergyConservation` to allow Specular-Workflow energy conservation to be turned off ([ColorDigital-PS](https://github.com/ColorDigital-PS)).
 
 ### ScreenshotTools
+
 - Added interface for argument `size` of screenshot methods ([Dok11](https://github.com/Dok11/))
 - Implementation usage of precision in combination height and width params ([Dok11](https://github.com/Dok11/))
 
 ### Sounds
+
 - Added `ISoundOptions.skipCodecCheck` to make `Sound` more flexible with URLs ([nbduke](https://github.com/nbduke))
 - Added `Scene.audioListenerPositionProvider` property, to enable setting custom position of audio listener ([Foxhoundn](https://github.com/foxhoundn))
 
 ### Sprites
+
 - SpritePackedManager extends SpriteManager so that a sprite sheet with different size sprites can be used ([JohnK](https://github.com/BabylonJSGuide))
 - MultiPickSprite and multiPickSpriteWithRay added to sprites ([JohnK](https://github.com/BabylonJSGuide))
 
+### WebXR
+
+- Compliance with the mozilla WebXR emulator ([RaananW](https://github.com/RaananW/))
+- Gamepad object is now exposed in the WebXRController class ([RaananW](https://github.com/RaananW/))
+- If canvas does not have WebXR support the scene will still render (mainly Firefox) ([RaananW](https://github.com/RaananW/))
+
 ### Ray
+
 - Added `Ray.intersectsAxis` to translate screen to axis coordinates without checking collisions ([horusscope](https://github.com/horusscope))
 
 ### GUI
+
 - Added `xmlLoader` to load GUI layouts from XML ([null0924](https://github.com/null0924))
 - Added `disableMobilePrompt` option to InputText for OculusQuest(and other android base VR devices) ([shinyoshiaki](https://github.com/shinyoshiaki))
 - Added `Button.delegatePickingToChildren` to let buttons delegate hit testing to embedded controls ([Deltakosh](https://github.com/deltakosh/))
@@ -118,20 +138,25 @@
 - Added `_getSVGAttribs` functionality for loading multiple svg icons from an external svg file via icon id.([lockphase](https://github.com/lockphase/))
 
 ### Particles
+
 - Added the feature `expandable` to the Solid Particle System ([jerome](https://github.com/jbousquie/))
 - Added the feature `removeParticles()` to the Solid Particle System ([jerome](https://github.com/jbousquie/))
 
 ### Navigation Mesh
+
 - Added moveAlong function to cast a segment on mavmesh ([CedricGuillemet](https://github.com/CedricGuillemet/))
 
 ### Node Material
+
 - Added Light intensity output to LightInformationBlock ([Drigax](https://github.com/drigax))
 
 ### Documentation
+
 - Added a note on shallow bounding of getBoundingInfo ([tibotiber](https://github.com/tibotiber))
 - Added a typo fix to the ArcRotateCamera setPosition method description ([schm-dt](https://github.com/schm-dt))
 
 ## Bug fixes
+
 - Fixed Textblock line spacing evaluation when linespacing > 0 ([Deltakosh](https://github.com/deltakosh/))
 - Fixed Xbox One gamepad controller button schemes ([MackeyK24](https://github.com/MackeyK24/))
 - Added support for `AnimationGroup` serialization ([Drigax](https://github.com/drigax/))
@@ -162,8 +187,10 @@
 - Fixed missing check in sceneTreeItemComponent resulting in gizmo to not end drag ([CedricGuillemet](https://github.com/CedricGuillemet))
 - Added missing callback triggers within texture loaders ([PierreLeBlond](https://github.com/PierreLeBlond))
 - Fixed `TextureLinkLineComponent` to no longer invert inspector-loaded textures ([Drigax](https://github.com/drigax))
+- Fixed a single frame drop after leaving webxr on some devices ([RaananW](https://github.com/RaananW/))
 
 ## Breaking changes
+
 - Setting mesh.scaling to a new vector will no longer automatically call forceUpdate (this should be done manually when needed) ([TrevorDev](https://github.com/TrevorDev))
 - `Tools.ExtractMinAndMaxIndexed` and `Tools.ExtractMinAndMax` are now ambiant functions (available on `BABYLON.extractMinAndMaxIndexed` and `BABYLON.extractMinAndMax`) ([Deltakosh](https://github.com/deltakosh/))
 - `Tools.QueueNewFrame` was removed in favor of `Engine.QueueNewFrame` ([Deltakosh](https://github.com/deltakosh/))

src/Cameras/XR/webXRCamera.ts

@@ -47,14 +47,14 @@ export class WebXRCamera extends FreeCamera {
     }
 
     /** @hidden */
-    public _updateForDualEyeDebugging(pupilDistance = 0.01) {
+    public _updateForDualEyeDebugging(/*pupilDistance = 0.01*/) {
         // Create initial camera rigs
         this._updateNumberOfRigCameras(2);
         this.rigCameras[0].viewport = new Viewport(0, 0, 0.5, 1.0);
-        this.rigCameras[0].position.x = -pupilDistance / 2;
+        // this.rigCameras[0].position.x = -pupilDistance / 2;
         this.rigCameras[0].outputRenderTarget = null;
         this.rigCameras[1].viewport = new Viewport(0.5, 0, 0.5, 1.0);
-        this.rigCameras[1].position.x = pupilDistance / 2;
+        // this.rigCameras[1].position.x = pupilDistance / 2;
         this.rigCameras[1].outputRenderTarget = null;
     }
 
@@ -69,29 +69,42 @@ export class WebXRCamera extends FreeCamera {
             return false;
         }
         var pose = xrSessionManager.currentFrame.getViewerPose(xrSessionManager.referenceSpace);
-        if (!pose || !pose.transform || !pose.transform.matrix) {
+        if (!pose) {
             return false;
         }
+        let globalTransform = false;
+        if (pose && pose.transform && pose.transform.matrix) {
+            globalTransform = true;
+            // Update the parent cameras matrix
+            Matrix.FromFloat32ArrayToRefScaled(pose.transform.matrix, 0, 1, WebXRCamera._TmpMatrix);
 
-        // Update the parent cameras matrix
-        Matrix.FromFloat32ArrayToRefScaled(pose.transform.matrix, 0, 1, WebXRCamera._TmpMatrix);
-        if (!this._scene.useRightHandedSystem) {
-            WebXRCamera._TmpMatrix.toggleModelMatrixHandInPlace();
+            if (!this._scene.useRightHandedSystem) {
+                WebXRCamera._TmpMatrix.toggleModelMatrixHandInPlace();
+            }
+            WebXRCamera._TmpMatrix.getTranslationToRef(this.position);
+            WebXRCamera._TmpMatrix.getRotationMatrixToRef(WebXRCamera._TmpMatrix);
+            Quaternion.FromRotationMatrixToRef(WebXRCamera._TmpMatrix, this.rotationQuaternion);
+
+            this.computeWorldMatrix();
         }
-        WebXRCamera._TmpMatrix.getTranslationToRef(this.position);
-        WebXRCamera._TmpMatrix.getRotationMatrixToRef(WebXRCamera._TmpMatrix);
-        Quaternion.FromRotationMatrixToRef(WebXRCamera._TmpMatrix, this.rotationQuaternion);
-        this.computeWorldMatrix();
 
         // Update camera rigs
         this._updateNumberOfRigCameras(pose.views.length);
         pose.views.forEach((view: any, i: number) => {
+            const currentRig = <TargetCamera> this.rigCameras[i];
             // Update view/projection matrix
-            Matrix.FromFloat32ArrayToRefScaled(view.transform.matrix, 0, 1, this.rigCameras[i]._computedViewMatrix);
-            Matrix.FromFloat32ArrayToRefScaled(view.projectionMatrix, 0, 1, this.rigCameras[i]._projectionMatrix);
+            if (!globalTransform && view.transform.position && view.transform.orientation) {
+                currentRig.position.copyFrom(view.transform.position);
+                currentRig.rotationQuaternion.copyFrom(view.transform.orientation);
+                currentRig.getViewMatrix(true);
+            } else {
+                Matrix.FromFloat32ArrayToRefScaled(view.transform.matrix, 0, 1, currentRig._computedViewMatrix);
+            }
+            Matrix.FromFloat32ArrayToRefScaled(view.projectionMatrix, 0, 1, currentRig._projectionMatrix);
+
             if (!this._scene.useRightHandedSystem) {
-                this.rigCameras[i]._computedViewMatrix.toggleModelMatrixHandInPlace();
-                this.rigCameras[i]._projectionMatrix.toggleProjectionMatrixHandInPlace();
+                currentRig._computedViewMatrix.toggleModelMatrixHandInPlace();
+                currentRig._projectionMatrix.toggleProjectionMatrixHandInPlace();
             }
 
             // Update viewport
@@ -99,14 +112,15 @@ export class WebXRCamera extends FreeCamera {
                 var viewport = xrSessionManager.session.renderState.baseLayer.getViewport(view);
                 var width = xrSessionManager.session.renderState.baseLayer.framebufferWidth;
                 var height = xrSessionManager.session.renderState.baseLayer.framebufferHeight;
-                this.rigCameras[i].viewport.width = viewport.width / width;
-                this.rigCameras[i].viewport.height = viewport.height / height;
-                this.rigCameras[i].viewport.x = viewport.x / width;
-                this.rigCameras[i].viewport.y = viewport.y / height;
+                currentRig.viewport.width = viewport.width / width;
+                currentRig.viewport.height = viewport.height / height;
+                currentRig.viewport.x = viewport.x / width;
+                currentRig.viewport.y = viewport.y / height;
             }
 
             // Set cameras to render to the session's render target
-            this.rigCameras[i].outputRenderTarget = xrSessionManager._sessionRenderTargetTexture;
+            currentRig.outputRenderTarget = xrSessionManager._sessionRenderTargetTexture;
+
         });
         return true;
     }

src/Cameras/XR/webXRController.ts

@@ -4,6 +4,7 @@ import { AbstractMesh } from "../../Meshes/abstractMesh";
 import { Matrix, Quaternion, Vector3 } from '../../Maths/math.vector';
 import { Ray } from '../../Culling/ray';
 import { Scene } from '../../scene';
+import { WebVRController } from '../../Gamepads/Controllers/webVRController';
 /**
  * Represents an XR input
  */
@@ -17,6 +18,14 @@ export class WebXRController {
      */
     public pointer: AbstractMesh;
 
+    private _gamepadMode = false;
+    /**
+     * If available, this is the gamepad object related to this controller.
+     * Using this object it is possible to get click events and trackpad changes of the
+     * webxr controller that is currently being used.
+     */
+    public gamepadController?: WebVRController;
+
     /**
      * Event that fires when the controller is removed/disposed
      */
@@ -48,6 +57,8 @@ export class WebXRController {
             if (this.parentContainer) {
                 this.parentContainer.addChild(this.grip);
             }
+        } else if (this.inputSource.gamepad) {
+            this._gamepadMode = true;
         }
     }
 
@@ -85,6 +96,12 @@ export class WebXRController {
                 this._tmpMatrix.decompose(this.grip.scaling, this.grip.rotationQuaternion!, this.grip.position);
             }
         }
+        if (this.gamepadController) {
+            // either update buttons only or also position, if in gamepad mode
+            this.gamepadController.isXR = !this._gamepadMode;
+            this.gamepadController.update();
+            this.gamepadController.isXR = true;
+        }
     }
 
     /**
@@ -103,12 +120,23 @@ export class WebXRController {
     }
 
     /**
+     * Get the scene associated with this controller
+     * @returns the scene object
+     */
+    public getScene() {
+        return this.scene;
+    }
+
+    /**
      * Disposes of the object
      */
     dispose() {
         if (this.grip) {
             this.grip.dispose();
         }
+        if (this.gamepadController && this._gamepadMode) {
+            this.gamepadController.dispose();
+        }
         this.pointer.dispose();
         this.onDisposeObservable.notifyObservers({});
     }

src/Cameras/XR/webXRControllerModelLoader.ts

@@ -3,6 +3,7 @@ import { WindowsMotionController } from '../../Gamepads/Controllers/windowsMotio
 import { OculusTouchController } from '../../Gamepads/Controllers/oculusTouchController';
 import { WebXRInput } from './webXRInput';
 import { ViveController } from '../../Gamepads/Controllers/viveController';
+import { WebVRController } from '../../Gamepads/Controllers/webVRController';
 
 /**
  * Loads a controller model and adds it as a child of the WebXRControllers grip when the controller is created
@@ -14,11 +15,12 @@ export class WebXRControllerModelLoader {
      */
     constructor(input: WebXRInput) {
         input.onControllerAddedObservable.add((c) => {
+            let controllerModel: WebVRController;
             if (c.inputSource.gamepad && c.inputSource.gamepad.id === "htc-vive") {
-                let controllerModel = new ViveController(c.inputSource.gamepad);
+                controllerModel = new ViveController(c.inputSource.gamepad);
                 controllerModel.hand = c.inputSource.handedness;
                 controllerModel.isXR = true;
-                controllerModel.initControllerMesh(c.grip!.getScene(), (m) => {
+                controllerModel.initControllerMesh(c.getScene(), (m) => {
                     m.isPickable = false;
                     m.getChildMeshes(false).forEach((m) => {
                         m.isPickable = false;
@@ -27,10 +29,10 @@ export class WebXRControllerModelLoader {
                     controllerModel.mesh!.rotationQuaternion = Quaternion.FromEulerAngles(0, Math.PI, 0);
                 });
             } else if (c.inputSource.gamepad && c.inputSource.gamepad.id === "oculus-touch") {
-                let controllerModel = new OculusTouchController(c.inputSource.gamepad);
+                controllerModel = new OculusTouchController(c.inputSource.gamepad);
                 controllerModel.hand = c.inputSource.handedness;
                 controllerModel.isXR = true;
-                controllerModel.initControllerMesh(c.grip!.getScene(), (m) => {
+                controllerModel.initControllerMesh(c.getScene(), (m) => {
                     controllerModel.mesh!.parent = c.grip!;
                     controllerModel.mesh!.rotationQuaternion = Quaternion.FromEulerAngles(0, Math.PI, 0);
                     controllerModel.mesh!.position.y = 0.034;
@@ -38,22 +40,23 @@ export class WebXRControllerModelLoader {
                 });
             } else if (c.inputSource.gamepad && c.inputSource.gamepad.id === "oculus-quest") {
                 OculusTouchController._IsQuest = true;
-                let controllerModel = new OculusTouchController(c.inputSource.gamepad);
+                controllerModel = new OculusTouchController(c.inputSource.gamepad);
                 controllerModel.hand = c.inputSource.handedness;
                 controllerModel.isXR = true;
-                controllerModel.initControllerMesh(c.grip!.getScene(), (m) => {
+                controllerModel.initControllerMesh(c.getScene(), (m) => {
                     controllerModel.mesh!.parent = c.grip!;
                     controllerModel.mesh!.rotationQuaternion = Quaternion.FromEulerAngles(Math.PI / -4, Math.PI, 0);
                 });
             } else {
-                let controllerModel = new WindowsMotionController(c.inputSource.gamepad);
+                controllerModel = new WindowsMotionController(c.inputSource.gamepad);
                 controllerModel.hand = c.inputSource.handedness;
                 controllerModel.isXR = true;
-                controllerModel.initControllerMesh(c.grip!.getScene(), (m) => {
+                controllerModel.initControllerMesh(c.getScene(), (m) => {
                     controllerModel.mesh!.parent = c.grip!;
                     controllerModel.mesh!.rotationQuaternion = Quaternion.FromEulerAngles(0, Math.PI, 0);
                 });
             }
+            c.gamepadController = controllerModel;
         });
     }
 }

src/Cameras/XR/webXRManagedOutputCanvas.ts

@@ -21,6 +21,12 @@ export class WebXRManagedOutputCanvas implements IDisposable {
      * @returns a promise that will resolve once the XR Layer has been created
      */
     public initializeXRLayerAsync(xrSession: any) {
+        // support canvases without makeXRCompatible
+        if (!(this.canvasContext as any).makeXRCompatible) {
+            this.xrLayer = new XRWebGLLayer(xrSession, this.canvasContext);
+            return Promise.resolve(true);
+        }
+
         return (this.canvasContext as any).makeXRCompatible().then(() => {
             this.xrLayer = new XRWebGLLayer(xrSession, this.canvasContext);
         });

src/Cameras/XR/webXRSessionManager.ts

@@ -38,6 +38,8 @@ export class WebXRSessionManager implements IDisposable {
     private _xrNavigator: any;
     private baseLayer: Nullable<XRWebGLLayer> = null;
 
+    private _sessionEnded: boolean = false;
+
     /**
      * Constructs a WebXRSessionManager, this must be initialized within a user action before usage
      * @param scene The scene which the session should be created for
@@ -69,9 +71,11 @@ export class WebXRSessionManager implements IDisposable {
     public initializeSessionAsync(xrSessionMode: XRSessionMode) {
         return this._xrNavigator.xr.requestSession(xrSessionMode).then((session: XRSession) => {
             this.session = session;
+            this._sessionEnded = false;
 
             // handle when the session is ended (By calling session.end or device ends its own session eg. pressing home button on phone)
             this.session.addEventListener("end", () => {
+                this._sessionEnded = true;
                 // Remove render target texture and notify frame obervers
                 this._sessionRenderTargetTexture = null;
 
@@ -118,6 +122,9 @@ export class WebXRSessionManager implements IDisposable {
         this.scene.getEngine().customAnimationFrameRequester = {
             requestAnimationFrame: this.session.requestAnimationFrame.bind(this.session),
             renderFunction: (timestamp: number, xrFrame: Nullable<XRFrame>) => {
+                if (this._sessionEnded) {
+                    return;
+                }
                 // Store the XR frame in the manager to be consumed by the XR camera to update pose
                 this.currentFrame = xrFrame;
                 this.onXRFrameObservable.notifyObservers(null);
@@ -141,7 +148,7 @@ export class WebXRSessionManager implements IDisposable {
         if (this.session) {
             this.session.end();
         }
-        return new Promise(() => {});
+        return new Promise(() => { });
     }
 
     /**
@@ -152,7 +159,7 @@ export class WebXRSessionManager implements IDisposable {
     public supportsSessionAsync(sessionMode: XRSessionMode) {
         if (!(navigator as any).xr || !(navigator as any).xr.supportsSession) {
             return Promise.resolve(false);
-        }else {
+        } else {
             return (navigator as any).xr.supportsSession(sessionMode).then(() => {
                 return Promise.resolve(true);
             }).catch((e: any) => {

src/Cameras/camera.ts

@@ -1205,7 +1205,7 @@ export class Camera extends Node {
 
     /**
      * Compute the world  matrix of the camera.
-     * @returns the camera workd matrix
+     * @returns the camera world matrix
      */
     public computeWorldMatrix(): Matrix {
         return this.getWorldMatrix();

src/Gamepads/Controllers/poseEnabledController.ts

@@ -231,9 +231,6 @@ export class PoseEnabledController extends Gamepad implements PoseControlled {
      * Updates the state of the pose enbaled controller and mesh based on the current position and rotation of the controller
      */
     public update() {
-        if (this.isXR) {
-            return;
-        }
         super.update();
         this._updatePoseAndMesh();
     }