Mixed Reality Image Display with Orientation-Based Delay Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional MR technologies experience delays in image processing and display due to the time required for rendering virtual images and subsequent processing, leading to discomfort for the user.
Innovation Solution
An image display device that includes an image sensor, orientation sensor, processors, and a display, which perform rendering, correction, and composition processes to reduce delay by using orientation information for real-time image correction and composition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If conventional MR technology performs rendering process for virtual images and subsequent processing, then virtual reality content can be displayed, but delay time from image capture to display increases causing user discomfort
Solution Approach 1:
The patent applies preliminary action by performing the rendering process for virtual images before the captured images are fully processed. Specifically, the rendering process generates virtual images in advance based on predicted head movement, allowing the composition of captured and rendered images to occur with minimal delay. This proactive rendering approach reduces the overall processing time and eliminates latency-related discomfort for the user.
2Adaptability or versatility
If rendering process is performed for virtual images, then virtual space can be displayed, but processing time increases causing delay
Solution Approach 1:
The rendering process is executed in advance before the captured images complete their processing pipeline. The system predicts the user's head movement and pre-renders virtual images that will be needed, thereby reducing the critical path processing time while maintaining full virtual space display capability.
Solution Approach 2:
The patent implements dynamic adjustment of rendering parameters based on detected head movement. The rendering process adapts its resolution, field of view, and complexity based on the user's actual movements, allowing high-quality virtual space display when needed while reducing processing load during stable periods, thus optimizing the balance between display capability and processing time.
3Manufacturing precision
If image processing and composition are performed sequentially, then image quality can be maintained, but frame rate decreases due to processing delays
Solution Approach 1:
The rendering of virtual images is performed in advance before the captured images are fully processed and ready for composition. This overlapping of processing stages allows the composition step to proceed immediately when captured images are available, maintaining image quality through proper processing while increasing frame rate by eliminating sequential waiting time.
Solution Approach 2:
The patent ensures continuous processing by overlapping the rendering process with the captured image processing pipeline. While captured images are being processed, virtual images are being rendered in parallel, and composition occurs continuously as soon as both inputs are ready. This continuous overlapping of useful actions maintains high image quality while maximizing frame rate throughput.
Data Source
AI summary
An image display device, on a basis of a position and orientation, which are determined from a first captured image and first orientation information corresponding to the first captured image when the first captured image has been captured, renders a first virtual image representing a virtual space as viewed from a viewpoint corresponding to the position and orientation, acquires a first corrected virtual image by correcting the first virtual image on a basis of second orientation information acquired after the first orientation information, acquires a composite image by combining a second captured image acquired after the first captured image with the first corrected virtual image, and displays the composite image on a display unit.