Depth-Aware Composite Image Rendering for Mixed Reality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for generating mixed reality images in head-mounted displays (HMDs) cause display delays and inaccurately represent the front-back positional relationship between real and translucent virtual objects due to ignoring the depth information of translucent objects.
Innovation Solution
The method involves acquiring separate layers for translucent and opaque virtual objects, along with real objects, and combining these layers based on depth information to generate a composite image that accurately represents the positional relationship.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the captured real image is converted into an object and processed by a rendering engine to determine front-back positional relationship, then the positional accuracy is improved, but the display delay time increases
Solution Approach 1:
The patent segments virtual objects into translucent and opaque categories, processing them through different pathways. Translucent objects are composited directly with depth information without full rendering engine processing, while opaque objects undergo complete rendering. This segmentation resolves the contradiction by reducing processing time for translucent objects while maintaining positional accuracy through depth information.
Solution Approach 2:
The patent performs preliminary acquisition of depth information for translucent objects before composite image generation. By pre-obtaining depth data and preparing translucent object layers in advance, the system reduces processing time during final image composition while ensuring accurate front-back positional relationships are established through pre-calculated depth information.
2Loss of time
If only virtual object images are generated by the rendering engine and combined with captured real images, then the display delay is reduced, but the front-back positional relationship accuracy deteriorates
Solution Approach 1:
The patent applies different processing qualities to different object types. Translucent objects receive localized processing that preserves depth information and transparency properties, while opaque objects receive complete rendering processing. This local quality differentiation allows reduced processing time for translucent objects without sacrificing positional relationship accuracy, as each object type receives appropriately tailored processing.
Solution Approach 2:
The patent introduces depth information as an additional dimension in the compositing process. By incorporating depth maps and processing images in multiple layers with depth-based ordering, the system achieves accurate front-back positional relationships without requiring complete rendering engine processing of all objects, thus reducing display delay while maintaining precision.
3Productivity
If translucent objects are processed without considering their depth information, then the processing speed is improved, but the representation accuracy of the composite image deteriorates
Solution Approach 1:
The patent extracts depth information specifically for translucent objects and separates it from the main rendering pipeline. By extracting depth data into a dedicated channel that is processed independently and applied during compositing, the system maintains high processing speed while ensuring depth information is preserved for accurate front-back positioning of translucent objects in the final composite image.
Data Source
AI summary
An information processing apparatus acquires a first layer including a first image in which a translucent object is drawn and first depth information that corresponds to the first image. The information processing apparatus acquires a second layer including a second image in which an opaque object is drawn and second depth information that corresponds to the second image. The information processing apparatus acquires a third layer including a third image in which a real object, which is placed in a real space, is drawn and third depth information that corresponds to the third image. The information processing apparatus draws a composite image in which the first image, the second image, and the third image are combined on a basis of the first depth information, the second depth information, and the third depth information.