Saliency-Based Video Resampling for Efficient Object Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks for video processing, particularly in resource-constrained devices, face challenges with high computational and power demands due to the processing of high-resolution video content, leading to inefficiencies and performance degradation when down-sampling is naively applied.
Innovation Solution
Implement saliency-based input resampling, where salient portions of video frames are sampled at higher resolution and less important regions are down-sampled, preserving fine-grained details while reducing computational load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high-resolution video frames are processed, then detection accuracy is improved, but computational cost and power consumption increase
Solution Approach 1:
The patent applies local quality by differentiating resolution requirements across different spatial regions of the video frame. Salient regions containing objects of interest are processed at high resolution to maintain detection accuracy, while non-salient background regions are down-sampled to reduce computational load and power consumption. This spatially adaptive approach ensures that computational resources are concentrated where they are most needed for accurate detection.
2Measurement precision
If high-resolution video frames are processed, then detection accuracy is improved, but device complexity increases
Solution Approach 1:
The system implements local quality by generating a saliency map that identifies regions requiring high-resolution processing. Only these identified salient regions are processed at full resolution through the neural network, while other regions are down-sampled. This dramatically reduces the number of pixels requiring complex neural network computations, thereby lowering device complexity while preserving detection accuracy for important regions.
Solution Approach 2:
The patent segments the video frame into salient and non-salient regions based on the saliency map. This segmentation allows differential processing where only the salient segments are processed at high resolution through the computationally intensive neural network, while non-salient segments are processed at lower resolution or excluded from detailed analysis, reducing overall computational complexity.
3Productivity
If down-sampling is applied to reduce computational load, then processing speed is improved, but detection accuracy deteriorates
Solution Approach 1:
The patent resolves this contradiction by applying down-sampling selectively only to non-salient regions while maintaining high resolution in salient regions. This ensures that processing speed is improved through reduced computational load in background areas, while detection accuracy is preserved in object-containing regions that require high-resolution processing.
Data Source
AI summary
A processor-implemented method of video processing using includes receiving, via an artificial neural network (ANN), a video including a first frame and a second frame. A saliency map is generated based on the first frame of the video. The second frame of the video is sampled based on the saliency map. A first portion of the second frame is sampled at a first resolution and a second portion of the second frame is sampled at a second resolution. The first resolution is different than the second resolution. A resampled second frame is generated based on the sampling of the second frame. The resampled second frame is processed to determine an inference associated with the video.