Adaptive Inference in Compressed Video Domain
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing adaptive action recognition techniques for video data require significant computational resources and memory, even when processing only selected frames or clips, due to the need for explicit temporal information processing in the uncompressed RGB domain, which is inefficient for devices with limited power and computational capacity.
Innovation Solution
The method processes compressed video data by reconstructing RGB images from motion and residual information, allowing for simultaneous use of all three modalities (RGB, motion vectors, and residuals) without a separate uncompressed source, and selects subsets of frames for further processing based on decision information generated from motion and residual analysis, using a decision module and inference model trained jointly as an end-to-end machine learning model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If adaptive action recognition techniques process video data using explicit temporal information in the uncompressed RGB domain, then the accuracy of action recognition is improved, but the computational resources and memory requirements increase significantly
Solution Approach 1:
The patent transitions from processing video data in the traditional uncompressed RGB domain to operating directly in the compressed domain, utilizing motion vectors and residuals as additional dimensional information. This dimensional shift allows the system to extract temporal information without requiring full uncompressed frame sequences, thereby reducing computational and memory requirements while maintaining action recognition accuracy.
Solution Approach 2:
The patent extracts only the essential temporal information needed for action recognition from the compressed video stream by utilizing motion vectors and residuals, rather than processing entire uncompressed frames. This extraction approach isolates and processes only the relevant temporal components, significantly reducing the computational burden and memory usage associated with handling complete frame sequences.
2Loss of time
If adaptive approaches select subsets of frames for processing, then the processing time is reduced, but the temporal information completeness may be compromised
Solution Approach 1:
The patent introduces motion vectors and residuals as intermediary elements that bridge the gap between selected frames. These intermediaries contain encoded temporal relationships that allow the system to infer motion and changes between frames without processing every intermediate frame, thus maintaining temporal information completeness while reducing processing time through selective frame processing.
Solution Approach 2:
The patent changes the representation parameters from full RGB frame data to compressed domain parameters (motion vectors and residuals). This parameter transformation allows the system to capture temporal dynamics with significantly reduced data volume, enabling efficient processing of frame subsets while preserving essential temporal information through the compressed representation.
Data Source
AI summary
Methods, devices and computer-readable media for processing a compressed video to perform an inference task are disclosed. Processing the compressed video may include selecting a subset of frame encodings of the compressed video, or zero or more modalities (RGB, motion vectors, residuals) of a frame encoding, for further processing to perform the inference task. Pre-existing motion vector and/or residual information in frame encodings of the compressed video are leveraged to adaptively and efficiently perform the inference task. In some embodiments, the inference task is an action recognition task, such as a human action recognition task.