XR Recording Pipeline with Separate Display and Capture Frame Rates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Challenging to record content displayed on head-mounted devices at a different frame rate than the display frame rate, as direct copying results in quality issues.
Innovation Solution
Implement a recording pipeline with a separate recording compositor that uses compositor descriptors to generate and record merged video frames at a different frame rate, optimizing recording quality while maintaining display performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If the displayed content is copied directly for recording, then the recording process is simple, but the recording quality cannot be optimized independently from display quality
Solution Approach 1:
The system separates the display pipeline and recording pipeline into independent processing paths. The display pipeline processes frames at display frame rate with display-specific parameters, while the recording pipeline processes frames at recording frame rate with recording-optimized parameters. This segmentation allows each pipeline to be optimized independently for its specific purpose without compromising the other.
Solution Approach 2:
The system changes key parameters between display and recording pipelines, including frame rate (display frame rate vs. recording frame rate), resolution (display resolution vs. recording resolution), and image correction parameters (compositor descriptors). These parameter changes enable the recording pipeline to achieve higher quality recordings by using optimized settings different from the display settings.
2Manufacturing precision
If the recording pipeline operates at the same frame rate as display, then synchronization is simple, but recording quality cannot be optimized independently
Solution Approach 1:
The system segments the frame processing into display frames and recording frames with different frame rates. Display frames are generated at display frame rate for real-time viewing, while recording frames are generated at recording frame rate for high-quality capture. The compositor descriptor mechanism bridges these segments by transferring necessary compositing information without requiring frame rate matching.
Solution Approach 2:
The compositor descriptor acts as an intermediary between the display pipeline and recording pipeline. It carries essential compositing information (such as virtual content positioning and image correction parameters) from the display pipeline to the recording pipeline, enabling the recording pipeline to reconstruct high-quality frames at its own frame rate without being constrained by display frame rate synchronization.
3Manufacturing precision
If high resolution is used for recording, then recording quality improves, but processing complexity and resource usage increase
Solution Approach 1:
The system performs preliminary compositing operations in the display pipeline and encodes the results in compositor descriptors before the recording pipeline processes frames. This preliminary action allows the recording pipeline to focus only on rendering and merging operations at high resolution, rather than performing full compositing from scratch, thereby reducing processing complexity while maintaining high recording quality.
Solution Approach 2:
The recording pipeline copies essential compositing information from the compositor descriptors generated by the display pipeline. Instead of independently computing all compositing parameters, the recording pipeline uses these copied parameters to efficiently reconstruct high-quality frames, reducing the computational burden of high-resolution processing.
Data Source
AI summary
A head-mounted device is provided that includes a variety of subsystems for generating extended reality content, displaying the extended reality content, and recording the extended reality content. The device can include a graphics rendering pipeline configured to render virtual content, tracking sensors configured to obtain user tracking information, a virtual content compositor configured to composite virtual frames based on the virtual content and the user tracking information, cameras configured to capture a video feed, a media merging compositor configured to overlay the composited virtual frames and the video feed to output merged video frames having a first frame rate for display, and a recording pipeline configured to record content having a second frame rate different than the first frame rate. The recording pipeline can record content exhibiting a higher quality than the content being displayed. A portion of the recorded content containing sensitive information can optionally be blurred.