XR Perception-of-Interest Rendering for Low-Latency Remote Maintenance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing remote interaction technologies in Extended Reality (XR) sessions lack effective communication of three-dimensional spatial relationships and fail to accurately render large or heavy models, leading to high latency and resource inefficiency, which hinders effective remote assistance in device maintenance.
Innovation Solution
A system and method for rendering a perception of interest in XR sessions by identifying and mapping user expertise with targeted components, using machine learning to unify and modify perceptions, and rendering them in real-time on head-mounted displays, while tracking user movements to enhance interaction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If the entire real-world scene is rendered in the virtual session, then complete spatial relationship information is provided, but resource usage increases and latency increases
Solution Approach 1:
The patent segments the virtual session rendering into two parts: (1) the entire real-world scene is captured and transmitted, and (2) only the perception of interest (specific component or area) is rendered with high fidelity in the virtual session. This segmentation allows complete spatial relationship information to be conveyed while reducing the computational resources needed for rendering the entire scene in virtual reality.
Solution Approach 2:
The patent extracts the perception of interest from the entire real-world scene and renders it separately with enhanced detail. By taking out only the relevant portion (the perception of interest) and rendering it in the virtual session, the system reduces overall resource consumption while maintaining complete spatial context through the transmitted real-world scene data.
2Reliability
If large and heavy models like aircraft and wind turbines are rendered, then complete device representation is achieved, but latency increases
Solution Approach 1:
The patent segments the large device models into two representation levels: (1) a complete but simplified model is transmitted to establish accurate spatial relationships, and (2) only the perception of interest (specific component) is rendered with high fidelity. This segmentation reduces the computational load for rendering large models while maintaining accurate device representation.
Solution Approach 2:
The patent applies local quality by rendering high-fidelity details only for the perception of interest (specific component or area) while using lower-fidelity representation for the rest of the large device model. This approach maintains accurate device representation where needed while reducing overall rendering latency.
3Loss of information
If the entire real-world scene is rendered, then complete spatial context is provided, but resource usage increases
Solution Approach 1:
The patent segments the rendering task into (1) transmission of the entire real-world scene for spatial context and (2) selective rendering of only the perception of interest. This segmentation preserves complete spatial context while improving rendering efficiency by focusing computational resources on the relevant portion only.
Solution Approach 2:
The patent extracts and renders only the perception of interest from the entire real-world scene. By taking out the relevant portion and rendering it separately, the system maintains complete spatial context through the transmitted scene data while significantly improving rendering efficiency.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Disclosed is a method (400) for rendering a perception of interest (172) in an Extended Reality (XR) session in real-time. The method (400) includes identifying first perceptions of an object (170) based on capturing the object (170) by a capturing device (180). The method (400) further includes determining one or more users (150) associated with the first perceptions based on correlating predefined users and the first perceptions. Further, the method (400) includes creating a unified perception by combining the first perceptions associated with the one or more users (150). The method includes determining the perception of interest (172) based on the unified perception. The perception of interest (172) is indicative of a view of a targeted component of the object (170). Further, the method (400) includes rendering the perception of interest (172) on a headmounted display (HMD) device (160) associated with each of the one or more users (150).