Mixed Reality Rendering With Real-Object Material Emulation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing mixed reality devices provide limited information about the real-world environment, leading to sub-optimal interaction between simulated and real-world objects, resulting in a less immersive experience.
Innovation Solution
A system and method that utilizes cameras and processors to capture and analyze real-world environments, identifying spatial geometry and material categories of objects, generating mixed-reality images and audio signals that emulate the material properties of real objects, enhancing the interaction and realism of the mixed reality experience.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If existing specialized devices capture basic geometric information of real-world objects, then the device complexity is reduced and processing is simplified, but the material property representation is insufficient leading to sub-optimal interaction between simulated and real objects
Solution Approach 1:
The system changes the parameters of image analysis from basic geometric extraction to include material property identification. By analyzing multiple image sequences and applying machine learning algorithms, the system extracts material categories and properties (such as texture, reflectivity, transparency) in addition to geometric information, thereby reducing information loss while managing processing complexity through intelligent algorithms
Solution Approach 2:
The system creates virtual copies of real-world objects that inherit and emulate the material properties of the captured objects. These virtual objects are generated by copying not only geometric data but also material characteristics from the analyzed image sequences, enabling realistic interactions between virtual and physical objects in the mixed reality environment
2Measurement precision
If the system analyzes multiple image sequences to identify material categories and spatial geometry, then the accuracy of object representation is improved, but the processing time increases
Solution Approach 1:
The system performs preliminary analysis of image sequences to pre-identify material categories and spatial geometry before generating the final mixed reality representation. By preparing and indexing the analyzed data in advance, the system achieves high measurement precision while reducing real-time processing requirements through efficient data retrieval and utilization
Solution Approach 2:
The system continuously processes image sequences in a streamlined manner, maintaining consistent analysis across multiple frames to accurately identify material properties and spatial relationships. This continuous processing approach ensures high accuracy while optimizing performance through efficient algorithms that maintain useful action throughout the capture and analysis process
Data Source
Figure 1A~1B
Figure 2
Figure 3A~4
AI summary
Disclosed is system (102) and method for enhancing user's immersion in mixed reality. Processor (108) controls camera (106, 300) to capture images of real-world environment; analyse images to identify spatial geometry of real objects and material categories to which real objects belong; process images to generate mixed-reality images, based upon spatial geometry and material category of at least one real object that is represented by at least one virtual object in mixed-reality images, wherein visual behaviour of virtual object emulates material property associated with material category of real object; and render mixed-reality images. Processor generates audio signal representative of acoustic behaviour of virtual object, based upon spatial geometry and material category of real object, wherein acoustic behaviour is to emulate material property associated with material category of real object; and plays the audio signal substantially simultaneously with rendering of mixed-reality images.