Post-Capture AR Content Editing via Layered Rendering Pipeline
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in efficiently processing and enhancing digital images, particularly in capturing and rendering augmented reality (AR) content using power and resource-constrained electronic devices, while reducing latency and power consumption.
Innovation Solution
The implementation of an extensible graphical rendering pipeline software architecture that supports layering of post-processing operations, enabling efficient processing and rendering of AR content on devices like smart glasses, which includes a messaging client application and an eyewear system that captures, processes, and displays AR content.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If post-processing operations are performed on captured AR content, then the quality and interactivity of AR media is improved, but the processing time and latency increase
Solution Approach 1:
The system performs preliminary encoding of captured video frames to an intermediate format during the capture phase, so that post-processing operations can be applied more efficiently later. This preliminary preparation reduces the computational burden during post-processing, thereby reducing latency while maintaining quality.
Solution Approach 2:
The post-processing pipeline is segmented into multiple independent stages (e.g., decoding, filtering, effects application, re-encoding). Each stage processes only specific aspects of the AR content, allowing for optimized processing at each step and reducing overall processing time compared to monolithic processing.
2Reliability
If complex post-processing operations are applied to enhance AR content, then the interactivity and quality of AR media is improved, but the power consumption increases
Solution Approach 1:
Video frames are preliminarily encoded to an intermediate format during capture, reducing the computational complexity of subsequent post-processing operations. This preliminary preparation significantly reduces power consumption during post-processing while maintaining the ability to apply quality-enhancing filters and effects.
Solution Approach 2:
The system changes the parameter of video encoding format dynamically - using a lower-complexity intermediate format during post-processing and transitioning to the final high-quality format only when necessary. This parameter change reduces computational complexity and power consumption during the most intensive processing phases.
3Reliability
If video frames are re-encoded after post-processing, then the quality of AR content is improved, but the processing time and power consumption increase
Solution Approach 1:
The system performs preliminary encoding to an intermediate format during capture, which serves as a foundation for efficient post-processing. This preliminary action avoids the need for complex re-encoding operations later, maintaining quality while improving processing efficiency.
Solution Approach 2:
Instead of re-encoding the entire video stream after post-processing, the system applies post-processing operations to individual video frames or segments that have already been encoded to an intermediate format. This copying approach to processing maintains quality while significantly improving efficiency compared to full re-encoding.
Data Source
AI summary
The subject technology receives, by a client device from a storage device, first image data captured by the client device at a previous time. The subject technology receives first metadata corresponding to at least a first image processing operation and a second image processing operation. The subject technology generates, in a first render pass, second image data based on the first metadata and the image processing operation performed on the first image data. The subject technology generates, in a second render pass, third image data based on the first metadata and the second image processing operation performed on the second image data. The subject technology generates second metadata comprising information corresponding to the third image data. The subject technology generates a composite AR content item comprising the second metadata, the third image data, and the first image data.