Mixed Reality CG Rendering Using Environment Map Position Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In mixed reality applications using video see-through type head-mounted displays, delays in displaying CG images relative to real space images lead to deviations in position, attitude, and optical characteristics, resulting in unnatural and inconsistent visual experiences.
Innovation Solution
An image processing apparatus that acquires information on the position and attitude of an imaging apparatus and associated image processing parameters from an environment map, allowing for the generation of CG images that maintain optical consistency with the real space by predicting the position and attitude of the imaging apparatus and adjusting image processing parameters accordingly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If CG image rendering is performed with high processing load to ensure quality, then image quality is improved, but display delay increases
Solution Approach 1:
The system performs preliminary actions by estimating position and attitude before actual CG image rendering. The position and attitude estimation is performed in advance using captured images, allowing the CG rendering to start from a pre-computed state, thereby reducing the overall processing time and display delay while maintaining image quality.
2Loss of time
If CG image is displayed without waiting for complete processing to reduce delay, then display time is reduced, but position and attitude deviation occurs
Solution Approach 1:
Position and attitude are estimated in advance before CG rendering completes. This preliminary estimation provides accurate spatial information that is then used during the rendering process, ensuring that even when CG image display is accelerated, the position and attitude remain accurate and consistent with the captured real space image.
Solution Approach 2:
The system uses feedback from the captured real space image to continuously refine and correct the position and attitude estimation. This feedback mechanism ensures that any deviations are detected and corrected, maintaining accuracy while allowing for reduced processing delays.
3Manufacturing precision
If image processing parameters are applied immediately to real space image, then real space image quality is improved, but CG image parameter application is delayed
Solution Approach 1:
Image processing parameters are estimated in advance based on the captured real space image before CG rendering completes. This preliminary parameter estimation allows the CG image to be rendered and displayed with appropriate parameters applied, eliminating the delay in parameter application while maintaining real space image quality.
Data Source
AI summary
An image processing apparatus includes one or more processors and/or circuitry configured to: perform a first acquiring processing to acquire information on a position and attitude of an imaging apparatus that captures an image of a real space; perform a second acquiring processing to acquire information on parameters on image processing, with the parameters being associated with the position and attitude of the imaging apparatus, from an environment map; and perform a generation processing to generate an image of an object to be displayed in the real space, based on the information on the position and attitude of the imaging apparatus, and the information on the parameters, wherein in the first acquiring processing, the position and attitude of the imaging apparatus is estimated from a captured image of the real space, by using the environment map.