Real-Time Image Effects Using Keyframe Tracking Compensation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computational intensity and resource limitations of complex computer vision schemes, such as convolutional neural networks, make real-time image processing on mobile devices impractical, particularly for generating image masks in video sequences.
Innovation Solution
A tracking-based imaging system processes keyframes using complex image processing schemes like CNN and non-keyframes using optical flow maps, asynchronously generating image masks to achieve real-time display.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If complex computer vision schemes like CNN are applied to process every frame in real-time, then image processing accuracy and effectiveness are improved, but processing speed and real-time capability deteriorate due to computational intensity
Solution Approach 1:
The patent segments the video sequence into keyframes and non-keyframes, applying complex CNN processing only to keyframes while using optical flow tracking for non-keyframes. This segmentation allows the system to maintain high processing accuracy where needed while achieving real-time performance overall.
Solution Approach 2:
Instead of applying the full complex CV scheme to every frame (excessive action), the patent applies it partially only to keyframes. The remaining frames are processed using a lighter optical flow-based approach, which is sufficient for maintaining temporal coherence and achieving real-time performance.
2Manufacturing precision
If complex CV schemes are executed on mobile devices with limited computational resources, then image processing capability is improved, but processing time increases making real-time operation impossible
Solution Approach 1:
The patent divides the processing workload by segmenting frames into keyframes (processed with CNN) and non-keyframes (processed with optical flow). This segmentation enables mobile devices to achieve real-time performance by not overloading the processor with intensive CNN operations on every frame.
Solution Approach 2:
The patent performs preliminary processing of keyframes with CNN to generate initial image masks, then uses optical flow to propagate and refine these masks across subsequent frames. This preliminary action reduces the computational burden on mobile devices while maintaining real-time capability.
3Speed
If asynchronous processing of image sequences is implemented, then real-time display capability is improved, but processing complexity and system coordination difficulty increase
Solution Approach 1:
The patent introduces an intermediary optical flow tracking mechanism that bridges the asynchronous processing of keyframes and non-keyframes. This intermediary allows the system to coordinate complex asynchronous operations by propagating image masks through temporal relationships, managing complexity through a structured intermediate step.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A mobile device can generate real-time complex visual image effects using asynchronous processing pipeline. A first pipeline applies a complex image process, such as a neural network, to keyframes of a live image sequence. A second pipeline generates flow maps that describe feature transformations in the image sequence. The flow maps can be used to process non-keyframes on the fly. The processed keyframes and non-keyframes can be used to display a complex visual effect on the mobile device in real-time or near real-time.