Mixed Reality Contact Detection Using Thickness Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current mixed reality technologies face challenges in accurately determining contact between virtual and real objects, particularly in estimating depth and thickness, leading to unnatural user interactions due to visible deviations and inability to consider object thickness, resulting in an unnatural user experience.
Innovation Solution
An information processing apparatus that acquires images, generates surface and thickness models of the user's operation unit, and determines contact with a computer graphics object based on these models, allowing for natural operation of virtual objects by overlaying real-time hand interactions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If depth estimation is made by cameras of a plurality of viewpoints, then depth of points for which association is possible can be estimated, but it is not possible to estimate the depth of an area that cannot be seen by all of the cameras
Solution Approach 1:
The patent introduces a time-of-flight camera as an intermediary measurement device to capture depth information for areas not visible to all viewpoint cameras. This mediator device fills the coverage gaps by providing depth data from regions that would otherwise be inaccessible to multi-viewpoint stereo matching, thereby resolving the contradiction between measurement precision and adaptability.
2Ease of manufacture
If a CG model is reconstructed from joint positions obtained by motion capture, then the model can be generated, but it is difficult to match the CG model and the position, thickness, or length of the real object, resulting in visible deviations
Solution Approach 1:
The patent changes the parameter set used for model reconstruction from joint positions alone to include both joint positions and depth information from time-of-flight cameras. This parameter expansion allows the CG model to accurately represent not only position but also thickness and length of real objects, eliminating visible deviations while maintaining ease of model generation through automated parameter acquisition.
3Device complexity
If only the object surface is modeled by using a distance measurement apparatus attached to the HMD, then the surface can be captured, but it is impossible to consider thickness, resulting in inability to make proper determination as to contact with the CG model
Solution Approach 1:
The patent transitions from two-dimensional surface modeling to three-dimensional volume modeling by integrating time-of-flight depth data. This dimensional enhancement allows the system to capture thickness information while maintaining device simplicity, as the time-of-flight camera provides automated depth measurement without requiring complex multi-camera setups or manual input, thereby improving contact determination reliability.
4Quantity of substance
If the resolution of the voxels is not made to be high in voxel-based depth model generation, then memory and time requirements are reduced, but the depth of the model cannot be estimated at good precision
Solution Approach 1:
The patent replaces the computationally intensive voxel-based depth model generation mechanism with a direct point cloud representation from time-of-flight camera data. This substitution eliminates the need for high-resolution voxel grids, significantly reducing memory and time resources while maintaining or improving depth model precision, as the time-of-flight data provides accurate depth measurements directly without requiring dense volumetric sampling.
Data Source
AI summary
An information processing apparatus that communicates with a display device that has an image capturing unit and that generates an image to be displayed on the display device, a composed image in which a CG (Computer Graphics) object in a virtual space according to the position and orientation of the image capturing unit is overlaid on a portion of an image, in real space, captured by the image capturing unit. An image generating unit performs a contact determination between the CG object in the virtual space and an operation unit in the real space based on a thickness model generated by a thickness model generating unit, and generates, as an image to be displayed on the display device, the composed image to which the result of the contact determination is reflected.