AR Video Preview Using Split Streams to Reduce Stuttering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Augmented reality (AR) effects in video processing can cause video stream stuttering due to high processing demands, particularly with machine learning-based effects, leading to undesirable user experiences in real-time rendering and recording.
Innovation Solution
Implementing two camera streams, one for real-time display with simplified AR effects and another for recording without effects, allowing for subsequent application of full AR effects to the recorded stream, thereby reducing processing load and stuttering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If full AR effects are applied to the camera stream in real-time, then the AR enhancement quality is improved, but the video stream experiences stuttering due to high processing demands
Solution Approach 1:
The video processing is segmented into two separate streams: a first camera stream processed with simplified AR effects for real-time display, and a second camera stream processed with full AR effects for recording. This segmentation allows the system to maintain smooth real-time rendering while still producing high-quality recorded video with complete AR effects.
2Productivity
If simplified AR effects are applied to the first camera stream, then the frame rate and smoothness are improved, but the AR enhancement quality is reduced
Solution Approach 1:
The system divides the video processing into two streams with different processing levels. The first stream uses simplified AR effects optimized for real-time display to maintain high frame rates, while the second stream applies full AR effects for high-quality recording. This resolves the contradiction by applying different processing strategies to different output requirements.
Solution Approach 2:
The patent creates a copy of the camera stream to process separately. The second camera stream serves as a copy that can be processed with full AR effects without impacting the real-time performance of the first stream, allowing high-quality AR enhancement without sacrificing frame rate in the primary display stream.
3Manufacturing precision
If full AR effects are applied to both display and recording simultaneously, then the AR quality is improved, but the processing load and device complexity increase
Solution Approach 1:
The processing load is segmented by creating two separate processing pipelines. The first pipeline handles real-time display with simplified effects requiring lower processing power, while the second pipeline handles recording with full effects. This segmentation allows the device to manage processing loads more efficiently while still delivering high-quality AR effects in the recorded output.
Solution Approach 2:
The system performs preliminary processing on the first camera stream with simplified AR effects for immediate display, while the second stream is processed with full effects subsequently for recording. This preliminary action approach allows the system to manage processing demands over time rather than requiring all processing to occur simultaneously at full capacity.
Data Source
AI summary
Image augmentation effects are provided on a device that includes a display and a camera. A simplified augmented reality effect is applied to a stream of images captured by the camera, to generate a preview stream of images. The preview stream of images is displayed on the display. A second stream of images corresponding to the first stream of images is saved to an initial video file. A full augmented reality effect, corresponding to the simplified augmented reality affect, is then applied to the second stream of images to generate a fully-augmented stream of images, which are saved to a further video file. The further video file can then be played back on the display to show the final, fully augmented reality effect as applied to the stream of images.