Asynchronous Image Processing Pipeline for Real-Time Mobile Effects
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computational intensity and resource limitations of complex computer vision schemes, such as convolutional neural networks, make real-time image processing on mobile devices impractical, especially for applications requiring 30 frames per second.
Innovation Solution
A tracking-based imaging system processes keyframes using complex image processing schemes like CNNs and non-keyframes using optical flow maps, allowing for real-time image effects by asynchronously generating image masks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If complex computer vision schemes like convolutional neural networks are applied to mobile devices, then image processing quality is improved, but processing time increases and real-time performance deteriorates
Solution Approach 1:
The patent segments the image processing task into two distinct pipelines: a first pipeline that processes keyframes (every Nth image) using computationally intensive machine learning schemes like convolutional neural networks, and a second pipeline that processes non-keyframes using faster optical flow-based tracking. This segmentation allows the system to maintain high processing quality for important frames while using more efficient methods for intermediate frames, thereby reducing overall processing time and enabling real-time performance.
2Measurement precision
If complex computer vision schemes are applied to mobile devices, then image processing quality is improved, but computational resources are exhausted
Solution Approach 1:
The patent divides the computational workload by processing only keyframes (every Nth image) through resource-intensive machine learning models, while using lighter optical flow-based algorithms for non-keyframes. This segmentation significantly reduces the total computational resources required compared to applying complex CV schemes to all frames, making real-time processing feasible on mobile devices with limited computational power.
Solution Approach 2:
Instead of applying complex machine learning schemes to all frames (excessive action), the patent applies these intensive schemes only to keyframes (partial action), which are sufficient for maintaining image processing quality. The remaining frames are processed using more efficient methods, achieving a balance between processing quality and computational resource consumption.
3Speed
If real-time image processing at 30 frames per second is implemented, then video smoothness is improved, but processing complexity increases
Solution Approach 1:
The patent segments the frame processing into two asynchronous pipelines operating at different complexities: the first pipeline handles keyframes with complex machine learning operations, while the second pipeline handles non-keyframes with simpler optical flow tracking. This segmentation enables the system to achieve 30 frames per second output by distributing computational complexity across multiple frames rather than concentrating it on every frame individually.
Solution Approach 2:
The patent maintains continuous real-time processing by ensuring that both pipelines operate continuously on incoming frames. The second pipeline continuously generates optical flow maps for non-keyframes, while the first pipeline continuously processes keyframes, creating an uninterrupted flow of processed frames that sustains 30 fps video output despite the varying complexity of individual processing operations.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A mobile device can generate real-time complex visual image effects using asynchronous processing pipeline. A first pipeline applies a complex image process, such as a neural network, to keyframes of a live image sequence. A second pipeline generates flow maps that describe feature transformations in the image sequence. The flow maps can be used to process non-keyframes on the fly. The processed keyframes and non-keyframes can be used to display a complex visual effect on the mobile device in real-time or near real-time.