No Miss Cache Structure for Real-Time Image Transformations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current mixed reality head-mounted display (HMD) systems face challenges in minimizing latency and bandwidth requirements due to the need for late stage reprojection adjustments, which can lead to user discomfort from image jitter and drift, especially when head movements are rapid.
Innovation Solution
The implementation of an optimally sized cache structure that fetches and manages image data to eliminate re-fetching from DRAM, minimizing bandwidth and power consumption by pre-fetching data before it is needed and evicting it when no longer required, while organizing data in a predictable order for efficient rendering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If late stage reprojection adjustments are applied to compensate for head movement, then image stability is improved, but DRAM bandwidth and power consumption increase
Solution Approach 1:
The system performs preliminary actions by pre-fetching image data into cache memory before it is needed for reprojection adjustments. This allows the cache to store multiple frames of image data in advance, so that when late stage reprojection is required, the data is already available in the cache rather than requiring additional DRAM bandwidth. The cache structure is optimally sized to hold sufficient data to support the predetermined range of reprojection adjustments without requiring re-fetching from DRAM.
2Manufacturing precision
If high resolution frames are rendered at high rate for LSR adjustments, then image quality is improved, but DRAM bandwidth requirements increase
Solution Approach 1:
The cache memory serves as an intermediary between the DRAM and the image processing pipeline. High resolution frame data is fetched from DRAM into the cache in advance, and then the cache supplies this data to the reprojection adjustment circuitry. This intermediary cache structure allows the system to maintain high image quality with high rate rendering while reducing the instantaneous DRAM bandwidth requirements, as the cache buffers the data flow and eliminates the need for continuous high-speed DRAM access during reprojection operations.
3Loss of time
If cache structure is used to pre-fetch image data, then latency is reduced, but cache size and complexity increase
Solution Approach 1:
The system optimizes the cache structure by carefully selecting and adjusting key parameters: the cache size is set to an optimal value that is sufficient to hold multiple frames of image data needed for reprojection adjustments, the pre-fetch timing is calibrated to load data into cache just before it is needed, and the cache replacement policy is configured to maintain the most recently accessed image data. By optimizing these parameters, the system achieves low latency without requiring an excessively large or complex cache structure.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
Systems and methods are disclosed herein for providing improved cache structures and methods that are optimally sized to support a predetermined range of late stage adjustments and in which image data is intelligently read out of DRAM and cached in such a way as to eliminate re-fetching of input image data from DRAM and minimize DRAM bandwidth and power. The systems and methods can also be adapted to work with compressed image data.