CGR Object Reflection Synthesis for Mixed Reality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer-generated reality (CGR) systems fail to accurately render reflective surfaces of CGR objects in physical environments due to limited camera views, resulting in incomplete reflections of surrounding objects that were not captured in the initial camera field.
Innovation Solution
The system synthesizes an environment map by determining textural information from observed camera images and generating synthesized texture for missing portions, allowing for a complete and realistic representation of the physical environment on reflective surfaces of CGR objects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If environment maps are generated from user-provided images of separate environments, then the reflective surfaces of CGR objects can be rendered, but the reflections do not accurately represent the actual physical objects surrounding the CGR object in the current scene
Solution Approach 1:
The system performs preliminary actions by capturing images of the physical environment before inserting the CGR object. These captured images are then used to generate an environment map that accurately represents the actual physical surroundings, ensuring that reflections are rendered with high accuracy and reliability for the specific physical environment.
2Loss of information
If only a limited camera field of view is used, then the camera system remains simple and manageable, but portions of physical objects that should be reflected in the CGR object are not captured
Solution Approach 1:
The system transitions from a two-dimensional camera field of view to a three-dimensional point cloud representation of the physical environment. By capturing depth information and converting images into a 3D point cloud, the system recovers surfaces that extend beyond the original camera field of view, ensuring complete reflection data without adding complex multi-camera systems.
3Loss of information
If the camera field of view is expanded to capture all surrounding surfaces, then complete reflections can be achieved, but the device complexity and processing requirements increase significantly
Solution Approach 1:
The system creates a digital copy of the physical environment in the form of a point cloud and environment map. This virtual representation captures all necessary reflection information without requiring physical expansion of the camera system. The copied environmental data can then be efficiently processed and applied to CGR object reflections, reducing the need for complex real-time processing of multiple camera feeds.
Data Source
AI summary
Various implementations disclosed herein include devices, systems, and methods that render a reflective surface of a computer-generated reality (“CGR”) object based on synthesis in a CGR environment. In order to render a reflective surface of the CGR object, one exemplary implementation involves synthesizing an environment map of a CGR environment representing a portion of a physical scene based on observed characteristics of the physical scene. In an implementation, generation of a complete environment map includes identifying pixels of the environment map with no corresponding texture and generating synthesized texture based on textural information associated with one or more camera images of the physical scene. In an implementation, a CGR object is rendered in the CGR environment, wherein an appearance of a reflective surface of the CGR object is determined based on the complete environment map of the CGR environment.