XR Image Stream Processing for Passthrough and Media Capture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In extended reality systems, point-of-view differences between the user and the camera can cause images displayed in passthrough mode to inaccurately reflect the user's physical environment, leading to user discomfort and a negative impact on the user experience.
Innovation Solution
A method and device that separately process an image stream to generate transformed images for display and stereo output images, using different transformation operations for each set, including perspective correction and image rectification, and generate metadata such as field of view and pose information to correct these differences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single transformation operation is applied to the image stream for both display and media capture, then the processing complexity is reduced, but the accuracy of the represented physical environment deteriorates causing user discomfort
Solution Approach 1:
The patent divides the image processing into two separate transformation pipelines: a first transformation operation for display output and a second transformation operation for media capture output. This segmentation allows each transformation to be optimized independently for its specific purpose, resolving the contradiction between processing simplicity and representation accuracy.
2Measurement precision
If different transformation operations are applied for display and media capture, then the accuracy of the physical environment representation is improved, but the processing complexity increases
Solution Approach 1:
The system maintains a single image stream capture pipeline that feeds both transformation operations, allowing the camera system to serve multiple functions (display and media capture) simultaneously. This universal input approach reduces the overall system complexity while enabling differentiated processing for each output purpose.
3Ease of operation
If perspective correction is applied to match user point-of-view, then user comfort is improved, but the fidelity of the captured media deteriorates
Solution Approach 1:
The patent applies different quality characteristics to different output streams: the display output receives perspective correction optimized for user comfort and natural perception, while the media capture output receives transformation optimized for geometric fidelity and accurate representation. Each output stream thus has locally optimized quality appropriate to its purpose.
4Manufacturing precision
If image rectification is applied for media capture, then the fidelity of captured media is improved, but user comfort in display mode deteriorates
Solution Approach 1:
The system dynamically selects and applies different transformation operations based on the intended output purpose. The first transformation operation (with perspective correction) is dynamically applied to display outputs to enhance user comfort, while the second transformation operation (with image rectification) is dynamically applied to media capture outputs to ensure fidelity. This dynamic adaptation resolves the contradiction between comfort and fidelity.
Data Source
AI summary
Embodiments disclosed herein are directed to devices, systems, and methods for separately processing an image stream to generate a first set of images for display by a set of displays and a second set of images for storage or transfer as part of a media capture event. Specifically, a first set of images from an image stream may be used to generate a first set of transformed images, and these transformed images may be displayed on a set of displays. A second set of images may be selected from the image stream in response to a capture request associated with a media capture event, and the second set of images may be used to generate a second set of transformed images. A set of output images may be generated from the second set of transformed images, and the set of output images may be stored or transmitted for later viewing.