Hand-Locked Virtual Object Rendering for Low-Lag AR Interaction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial reality systems struggle to efficiently render virtual objects in relation to physical objects, particularly hands, due to input and display lag, leading to obsolete frames and limited rendering rates, which complicates smooth interaction and immersion.
Innovation Solution
A system that employs hand-locked rendering techniques, using sub-frame updates based on hand and head pose tracking, allowing virtual objects to be anchored to specific points on the user's hand, with specialized APIs for applications to specify anchor points and interact with callback functions, and incorporates occlusion handling to ensure only visible portions of virtual objects are displayed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional rendering methods are used in artificial reality systems, then the system can display virtual objects, but input and display lag causes frames to become obsolete and rendering rates are limited
Solution Approach 1:
The patent segments the rendering process into multiple independent components: hand pose tracking, head pose tracking, virtual object generation, and image rendering. By processing these separately and combining them, the system reduces overall processing time and eliminates bottlenecks that cause lag and frame obsolescence.
Solution Approach 2:
The system performs preliminary actions by pre-generating virtual objects based on hand pose data before the actual rendering occurs. This allows the virtual objects to be ready in advance, reducing the time required during the actual rendering phase and preventing frame obsolescence due to processing delays.
2Measurement precision
If virtual objects are dynamically updated to match hand movements, then tracking fidelity improves, but computational complexity and processing time increase
Solution Approach 1:
The system uses the device's own sensors (cameras and tracking systems) to capture hand and head poses, and automatically processes this data to generate and render virtual objects. This self-service approach eliminates the need for external complex processing systems while maintaining high tracking fidelity.
Solution Approach 2:
The patent changes key parameters by processing hand pose data and head pose data independently at high rates, then combining them during rendering. This parameter separation allows the system to maintain high tracking fidelity for hand movements without proportionally increasing overall system complexity.
3Productivity
If high frame rates are achieved through sub-frame rendering, then immersion and interaction smoothness improve, but processing requirements and system complexity increase
Solution Approach 1:
The system implements periodic action by updating hand pose tracking and virtual object generation at specific intervals (sub-frames) rather than continuously. This allows the system to achieve high effective rendering rates while processing data periodically, reducing overall processing requirements and system complexity.
4Ease of operation
If virtual objects are anchored to hand positions, then user interaction becomes more intuitive, but occlusion handling and rendering complexity increase
Solution Approach 1:
The patent introduces an intermediary virtual hand model that represents the user's actual hand in the virtual environment. This intermediary serves as a mediator between the real hand and virtual objects, simplifying occlusion handling by allowing the system to render virtual objects relative to the virtual hand model rather than directly to the complex geometry of the actual hand.
Data Source
Figure 1A~1B
Figure 2
Figure 3
AI summary
In one embodiment, a method includes, by one or more computing devices, determining, based on first tracking data, a first viewpoint of the user and a first hand pose of the user. The method may include generating a virtual object in a virtual environment based on the first hand pose and a predetermined spatial relationship between the virtual object and the hand of the user. The method may include rendering a first image of the virtual object as viewed from the first viewpoint. The method may include determining, based on second tracking data, a second viewpoint of the user and a second hand pose. The method may include adjusting the first image of the virtual object based on changes from the first hand pose to the second hand pose. The method may include rendering and displaying a second image from the adjusted first image viewed from the second viewpoint.