Mixed Reality Object Segmentation and Virtual Peripheral Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In mixed reality environments, existing technologies face challenges in accurately tracking hand movements and aligning virtual and physical elements, leading to inconsistent interactive experiences and potential motion sickness due to processing delays and misalignment of images.
Innovation Solution
A method and system that captures images of physical interactions using a head-mounted display (HMD), measures the movement of the HMD, processes the images to generate composite images by shifting image data within a frame buffer, and displays virtual interactions, ensuring accurate alignment and reducing latency effects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If image processing is performed to generate composite images in mixed reality environments, then the alignment between virtual and physical elements is improved, but processing delays increase causing motion sickness
Solution Approach 1:
The patent applies preliminary action by pre-calculating and storing transformation matrices for different head positions and orientations before they are needed. When the HMD moves, the system retrieves pre-computed matrices rather than calculating them in real-time, significantly reducing processing delay while maintaining alignment accuracy between virtual and physical elements
Solution Approach 2:
The system implements dynamics by continuously updating transformation matrices based on real-time head movement data from sensors. The image processing pipeline dynamically adjusts the composite image generation parameters according to the current head position and orientation, ensuring accurate alignment while optimizing processing speed through adaptive frame skipping and resolution adjustment
2Stability of the object's composition
If real-time image processing is performed to update composite images during HMD movement, then consistency of virtual and physical elements is improved, but computational complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the image processing task into distinct stages: capturing the physical environment, segmenting virtual elements to be overlaid, generating transformation matrices based on head pose, and composing the final composite image. This modular approach allows each stage to be optimized independently, reducing overall computational complexity while maintaining consistency
Solution Approach 2:
The system uses copying by creating simplified representations of the physical environment and virtual elements as separate layers. Instead of processing the entire complex scene in real-time, the system copies essential geometric and visual properties into optimized data structures that can be rapidly transformed and combined, reducing computational load while preserving visual consistency
Data Source
AI summary
Methods and systems for providing a mixed reality (MR) interaction are provided. In one example, the method comprises: capturing a first image of a user's appendage and a physical peripheral input device, wherein the user's appendage covers at least a part of the physical peripheral input device; performing a segmentation of the first image into a first region of pixels corresponding to the user's appendage, wherein a second region of pixels corresponds to the physical peripheral input device; generating a composite image from the first image based on rendering, in place of the second region of pixels, pixels representing a virtual peripheral input device, such that the virtual peripheral input device appears in the composite image as at a same spatial location as the physical peripheral input device and covered at least partially by the user's appendage; and displaying the composite image to the user.