Compressed Video Object Detection via I-Frame Feature Interpolation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current computer vision algorithms require decompressing and processing uncompressed video frames, which is computationally expensive and impractical for resource-constrained devices like smartphones and security cameras, especially for real-time video analysis.
Innovation Solution
The approach involves directly processing compressed video frames using a two-stage object detection framework that performs heavy-duty feature extraction on I-frames and interpolates features for P- and B-frames using motion vectors, reducing computational load while maintaining accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If video frames are decompressed and processed frame-by-frame using conventional vision algorithms, then object detection accuracy is improved, but computational cost and processing time increase significantly
Solution Approach 1:
The patent segments the video processing task by type of frame: I-frames undergo full decompression and heavy-duty feature extraction, while P- and B-frames use lightweight feature extraction with motion compensation. This segmentation allows the system to maintain high accuracy on key frames while dramatically reducing processing load on redundant frames, resolving the contradiction between accuracy and processing speed.
Solution Approach 2:
The patent applies partial action by performing complete feature extraction only when necessary (on I-frames) and using simplified feature extraction with motion compensation for other frames. This partial processing approach maintains sufficient detection accuracy while reducing overall computational cost, directly addressing the speed-accuracy tradeoff.
2Measurement precision
If video frames are decompressed and processed frame-by-frame using conventional vision algorithms, then object detection accuracy is improved, but memory and storage requirements increase
Solution Approach 1:
The patent extracts only the essential feature information needed for object detection from I-frames, and then uses motion compensation to derive features for P- and B-frames without full decompression. This extraction approach obtains sufficient detection accuracy while minimizing memory and storage requirements by avoiding retention of full uncompressed frame data.
Solution Approach 2:
The patent creates simplified copies of feature information from I-frames and propagates them to P- and B-frames using motion compensation. Instead of storing and processing full uncompressed frames for every video frame, the system uses these derived feature copies, reducing memory and storage needs while maintaining detection capability.
3Productivity
If frame sampling is used to reduce processing load, then processing speed is improved, but detection accuracy deteriorates due to missed objects
Solution Approach 1:
The patent performs preliminary heavy-duty feature extraction on I-frames, which occur at regular intervals throughout the video. This preliminary processing on key frames provides a foundation that can be propagated to intermediate frames via motion compensation, ensuring that important objects are detected on I-frames while reducing the need for processing every single frame, thus improving speed without sacrificing accuracy.
Data Source
AI summary
Various embodiments of a framework which allow, as an alternative to resource-taxing decompression, efficient computation of feature maps using a compressed content data subset, such as video, by exploiting the motion information, such as a motion vector, present in the compressed video. This framework allows frame-specific object recognition and action detection algorithms to be applied to compressed video and other media files by executing only on I-frames in a Group of Pictures and linearly interpolating the results. Training and machine learning increases recognition accuracy. Yielding significant computational gains, this approach accelerates frame-wise feature extraction I-frame/P-frame/P-frame videos as well as I-frame/P-frame/B-frame videos. The present techniques may also be used for segmentation to identify and label respective regions for objects in a video.