Heterogeneous XR Interaction via Pose Transform Mirroring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users with non-extended reality (XR) capable devices, such as smartphones, are limited in participating in holocalls as they cannot render changes in pose within XR environments, hindering face-to-face interactions with users of XR-capable devices.
Innovation Solution
A system that enables users of non-XR capable devices to change their viewpoint within an XR environment based on gestural movements, synchronizing with XR-capable devices, allowing for a seamless holocall experience by receiving and transmitting 3D mesh data and pose transforms, effectively mirroring the XR experience across heterogeneous devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If users with non-XR capable devices participate in holocalls, then accessibility and user base are improved, but the ability to render pose changes and provide immersive XR experience deteriorates
Solution Approach 1:
The system creates a virtual copy of the non-XR device user within the XR environment by compositing video of the user into the 3D mesh environment using pose transforms. This virtual representation allows the non-XR user to be seen and interacted with in the XR space without requiring the physical device to have XR rendering capabilities.
Solution Approach 2:
The system introduces an intermediary processing layer that translates gestures from the non-XR device into pose transforms that control the virtual representation in the XR environment. This intermediary mechanism enables communication between devices with different capabilities without requiring the non-XR device to directly render XR content.
2Adaptability or versatility
If heterogeneous devices are used for holocalls, then device compatibility and user choice are improved, but synchronization of view perspective and pose transforms deteriorates
Solution Approach 1:
The system transforms gesture parameters from the non-XR device into pose transform parameters suitable for the XR environment. By changing the parameter representation and interpretation, the system enables accurate synchronization between heterogeneous devices with different input modalities and coordinate systems.
Solution Approach 2:
The system implements bidirectional feedback loops where pose transforms are exchanged between devices to maintain synchronized views. Each device receives and applies pose transform data from the other device, creating a closed-loop system that continuously adjusts and maintains synchronization accuracy despite device heterogeneity.
3Ease of operation
If video compositing with pose transforms is implemented, then immersive interaction quality is improved, but computational complexity and processing requirements deteriorate
Solution Approach 1:
The system segments the processing tasks between devices, with each device handling specific aspects of video compositing and pose transformation. This division of labor reduces the computational burden on individual devices while maintaining overall interaction quality through coordinated processing.
Data Source
AI summary
Extended reality interactions include capturing, with a first device, video of a first user and conveying same to a second, heterogeneous device. A 3D mesh is received by the first device from the second device for rendering an extended reality environment, which is simultaneously displayed on the second device. Video of a second user and pose transforms for compositing the video of the second user in the extended reality environment displayed on the first device is received. A view perspective of the video of the second user composited in the extended reality environment is based on the pose transforms. Input to the first device changes the view perspective. View perspective data is conveyed from the first device to the second device that causes a corresponding change in view perspective of the video of the first user composited in the extended reality environment simultaneously displayed on the second device.