Selective Visual Data Upsampling for AR Pass-Through Displays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing augmented and virtual reality systems face challenges in efficiently processing and displaying high-quality visual data, leading to increased hardware requirements and processing loads.
Innovation Solution
The implementation of selective upsampling of visual sensor data, specifically using foveated displays and hierarchical mipmaps, allows for targeted high-quality visuals in areas of focus while maintaining efficient resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If full-resolution visual data is processed and displayed across the entire display, then visual quality is improved, but system resource efficiency deteriorates
Solution Approach 1:
The patent applies local quality by differentiating visual quality across different regions of the display. The foveated display technique renders high-resolution visuals only in the foveal region where users focus their attention, while peripheral regions receive lower-resolution rendering. This resolves the contradiction by maintaining high visual quality where needed while improving system resource efficiency through selective resolution reduction in less critical areas.
Solution Approach 2:
The patent implements partial action by processing and rendering only a portion of the visual scene at full resolution rather than the entire scene. By identifying and prioritizing the foveal region, the system applies full processing resources selectively to the most visually important area, achieving high visual quality in critical regions while conserving system resources overall.
2Manufacturing precision
If high-resolution visual data is generated and processed, then display quality is improved, but processing load increases
Solution Approach 1:
The patent reduces processing load by applying high processing quality locally only to the foveal region rather than uniformly across the entire display. The rendering system calculates which regions require high-quality processing based on eye tracking data, then concentrates computational resources on those specific areas, thereby maintaining high display quality where perceived while significantly reducing overall processing load.
Solution Approach 2:
The patent implements dynamic resource allocation by continuously adjusting which regions receive high-resolution processing based on real-time eye tracking feedback. As the user's gaze moves, the system dynamically shifts the high-quality rendering focus to the new foveal region, optimizing processing load adaptively rather than statically, thus maintaining high display quality while minimizing sustained processing requirements.
3Manufacturing precision
If high-resolution visual data is captured and processed, then user experience is improved, but hardware requirements increase
Solution Approach 1:
The patent reduces hardware requirements by implementing local quality enhancement through selective upscaling. Instead of requiring the entire visual pipeline to operate at maximum resolution, the system captures or generates content at lower resolution and applies intelligent upscaling algorithms specifically to the foveal region, achieving high user experience quality in critical areas while reducing the hardware specifications needed for capture and processing.
Solution Approach 2:
The patent introduces an intermediary processing stage between low-resolution capture and high-resolution display. The upscaling algorithm acts as an intermediary that selectively enhances resolution in the foveal region using eye tracking information, allowing the system to use lower-specification hardware for capture and processing while still delivering high-quality visuals to the user through computational enhancement.
Data Source
AI summary
An efficiency manager can selectively upsample portions of captured data, for example to target certain portions of a user's field of view (e.g., pass-through display) for higher quality visuals. For example, the efficiency manager can select, based on user eye tracking, portions of the captured visual data for upsampling and omit upsampling for other portions. The efficiency manager can implement the selective upsampling using a hierarchical mipmap. A mipmap can store visual data at different quality levels. The efficiency manager can sparsely populate a high quality level of the mipmap with the selectively upsampled visual data. Rendering techniques (e.g., graphical processor unit hardware and/or rendering pipelines) can utilize mipmaps to efficiently render visual data for display.